GeoIP2 updater might try to download new file before it exists #18427
Comments
|
This night, our job tried to unzip until 5AM GMT+1. Maybe we can simply skip the monthly job on 2nd of each month? 🤔 |
|
Same here, https://download.db-ip.com/free/dbip-city-lite-2023-11.mmdb.gz returns a 404 at the time of writing. Though it will probably work in a few hours... Moving the cron to the 2nd of the month sound like a good idea to limit those issue without having data that are too stale. |
|
I created #21468 as a possible fix for this issue. |
When geolocation database updates are configured to be done monthly, they were scheduled on the 3rd day of _the week_, so Wednesday. For months that start on a Wednesday, that means that we tried to fetch a file that might not exist yet. We now schedule on the 3rd day of _the month_, as I believe was the original intent of dc98a97 that introduced the behavior a long time ago. So this gives 3 days to db-ip.com to update their files in all cases. Fixes #18427
|
Reopening, as the PR only adjusted the day the download is tried. We still should aim to implement a proper handling when the download fails. |
|
There is no proper handling if the file is simply missing. I think an administrator notification in GUI and an error logged (already exists), they are enough. |
When geolocation database updates are configured to be done monthly, they were scheduled on the 3rd day of _the week_, so Wednesday. For months that start on a Wednesday, that means that we tried to fetch a file that might not exist yet. We now schedule on the 3rd day of _the month_, as I believe was the original intent of dc98a97 that introduced the behavior a long time ago. So this gives 3 days to db-ip.com to update their files in all cases. Fixes #18427
When geolocation database updates are configured to be done monthly, they were scheduled on the 3rd day of _the week_, so Wednesday. For months that start on a Wednesday, that means that we tried to fetch a file that might not exist yet. We now schedule on the 3rd day of _the month_, as I believe was the original intent of dc98a97 that introduced the behavior a long time ago. So this gives 3 days to db-ip.com to update their files in all cases. Fixes #18427
When geolocation database updates are configured to be done monthly, they were scheduled on the 3rd day of _the week_, so Wednesday. For months that start on a Wednesday, that means that we tried to fetch a file that might not exist yet. We now schedule on the 3rd day of _the month_, as I believe was the original intent of dc98a97 that introduced the behavior a long time ago. So this gives 3 days to db-ip.com to update their files in all cases. Fixes #18427
reported in https://forum.matomo.org/t/geoip2autoupdater-failed-to-unzip-the-downloaded-file-is-not-a-valid-geolocation-database/43925 (and also found in my cronjob)
Expected Behavior
Whenever a GeoIP2 update job falls at the start of a month it might be possible that the
dbip-city-lite-2021-12.mmdb.gzdoesn't exist on the servers and the update failsCurrent Behavior
Possible Solution
In case a 404 is returned, Matomo could fetch the previous months file again or reschedule the job for a few hours later.
The text was updated successfully, but these errors were encountered: