Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GeoIPAutoUpdater fails with "failed to unzip '/var/www/stat/misc/GeoIPCity.dat.dat.gz' #9544

Closed
mattab opened this issue Jan 18, 2016 · 13 comments
Labels
Bug For errors / faults / flaws / inconsistencies etc. wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it.

Comments

@mattab
Copy link
Member

mattab commented Jan 18, 2016

GeoIPAutoUpdater: failed to unzip '/var/www/stat/misc/GeoIPCity.dat.dat.gz' after downloading 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz': The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.

As reported in http://forum.piwik.org/t/geoipautoupdater-error/16143

@mattab mattab added the Bug For errors / faults / flaws / inconsistencies etc. label Jan 18, 2016
@mattab
Copy link
Member Author

mattab commented Jan 18, 2016

Notes

  • This is reproducible on demo and demo2. I didn't notice but we did get email notification errors from the cron
  • that there is twice .dat in the filename, which maybe indicates an error (or maybe not?) failed to unzip '/var/www/stat/misc/GeoIPCity.dat.dat.gz.

Full error trace is:

ERROR [2016-01-06 00:56:32] /home/piwik-demo/www/demo.piwik.org/plugins/UserCountry/GeoIPAutoUpdater.php(161): GeoIPAutoUpdater: failed to unzip '/home/piwik-demo/www/demo.piwik.org/misc/GeoIPCity.dat.dat.gz' after downloading 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz': The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.
#0 /home/piwik-demo/www/demo.piwik.org/plugins/UserCountry/GeoIPAutoUpdater.php(96): Piwik\Plugins\UserCountry\GeoIPAutoUpdater->downloadFile('loc', 'http://geolite....')
#1 [internal function]: Piwik\Plugins\UserCountry\GeoIPAutoUpdater->update(NULL)
#2 /home/piwik-demo/www/demo.piwik.org/core/Scheduler/Scheduler.php(227): call_user_func(Array, NULL)
#3 /home/piwik-demo/www/demo.piwik.org/core/Scheduler/Scheduler.php(124): Piwik\Scheduler\Scheduler->executeTask(Object(Piwik\Plugins\UserCountry\GeoIPAutoUpdater))
#4 /home/piwik-demo/www/demo.piwik.org/plugins/CoreAdminHome/API.php(55): Piwik\Scheduler\Scheduler->run()
#5 /home/piwik-demo/www/demo.piwik.org/core/CronArchive.php(529): Piwik\Plugins\CoreAdminHome\API->runScheduledTasks()
#6 /home/piwik-demo/www/demo.piwik.org/core/CronArchive.php(283): Piwik\CronArchive->runScheduledTasks()
#7 /home/piwik-demo/www/demo.piwik.org/core/Access.php(466): Piwik\CronArchive->Piwik\{closure}()
#8 /home/piwik-demo/www/demo.piwik.org/core/CronArchive.php(285): Piwik\Access::doAsSuperUser(Object(Closure))
#9 /home/piwik-demo/www/demo.piwik.org/plugins/CoreConsole/Commands/CoreArchiver.php(27): Piwik\CronArchive->main()
#10 /home/piwik-demo/www/demo.piwik.org/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\Plugins\CoreConsole\Commands\CoreArchiver->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /home/piwik-demo/www/demo.piwik.org/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /home/piwik-demo/www/demo.piwik.org/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application->doRunCommand(Object(Piwik\Plugins\CoreConsole\Commands\CoreArchiver), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 [internal function]: Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /home/piwik-demo/www/demo.piwik.org/core/Console.php(79): call_user_func(Array, Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /home/piwik-demo/www/demo.piwik.org/core/Access.php(466): Piwik\Console->Piwik\{closure}()
#16 /home/piwik-demo/www/demo.piwik.org/core/Console.php(80): Piwik\Access::doAsSuperUser(Object(Closure))
#17 /home/piwik-demo/www/demo.piwik.org/vendor/symfony/console/Symfony/Component/Console/Application.php(126): Piwik\Console->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /home/piwik-demo/www/demo.piwik.org/misc/cron/archive.php(57): Symfony\Component\Console\Application->run()
#19 {main}

@mattab mattab added this to the 2.16.0 milestone Jan 18, 2016
@mattab
Copy link
Member Author

mattab commented Jan 18, 2016

Adding to 2.16.0 - @tsteur @sgiehl maybe you could take a look at this one? It started occuring in November and seems to be experienced by all users.

@tsteur
Copy link
Member

tsteur commented Jan 18, 2016

FYI: geoip.loc_db_url in option table in DB says http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz on demo. Had a quick look but code is almost impossible to understand so needs proper investigating or refactoring.

@hpvd
Copy link

hpvd commented Jan 18, 2016

in our installation (2.16b3) in backend there is no error message visible.
Looks "working ok"
=> is it ok in this installation (and does not appear everywhere)?
=> or should one add a error massage to report if there is a problem?
please see attachement

2016-01-18_23h06_34

@mattab mattab modified the milestones: 2.16.1, 2.16.0 Jan 27, 2016
@mattab mattab modified the milestones: 2.16.1, 3.0.0 Feb 3, 2016
@clauso
Copy link

clauso commented Feb 3, 2016

I have the same error today (I think I have seen it in the past):

ERROR [2016-02-03 00:05:11] /var/www/piwik/plugins/UserCountry/GeoIPAutoUpdater.php(161): GeoIPAutoUpdater: failed to unzip '/var/www/piwik/misc/GeoIPCity.dat.dat.gz' after downloading 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz': The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.

We are running piwik 2.15.0

I have this md5sum for the downloaded file:

$> md5sum /var/www/piwik/misc/GeoLiteCity.dat
97752fa1e1407d9c2c48008a4b9428c1  /var/www/piwik/misc/GeoLiteCity.dat

@nd2s
Copy link

nd2s commented Mar 5, 2016

Fails here as well. v2.16.0 on FreeBSD.

@clauso
Copy link

clauso commented Jul 13, 2016

Still get those mails about failures so I tried to look into it. I found out that I can trigger an update run manually by calling

php /var/www/piwik/console scheduled-tasks:run --force

A maybe interesting thing is that here I get an additional error that was not in the mails, see line 2:

INFO [2016-07-13 04:59:50] GeoIPAutoUpdater: successfully downloaded 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz'
INFO [2016-07-13 04:59:50] GeoIPAutoUpdater: Encountered PHP error when testing newly downloaded GeoIP database: 2: include(): Failed opening '/var/www/piwik/tmp/cache/tracker/piwikcache_Translations-en-e21bf04aeec884480f77854677fd2bf5fbc090f3.php' for inclusion (include_path='/var/www/piwik/vendor/pear/pear_exception:/var/www/piwik/vendor/pear/console_getopt:/var/www/piwik/vendor/pear/pear-core-minimal/src:/var/www/piwik/vendor/pear/archive_tar:.:/usr/share/php') on line 112 of /var/www/piwik/vendor/doctrine/cache/lib/Doctrine/Common/Cache/PhpFileCache.php.
ERROR [2016-07-13 04:59:50] /var/www/piwik/plugins/UserCountry/GeoIPAutoUpdater.php(161): GeoIPAutoUpdater: failed to unzip '/var/www/piwik/misc/GeoIPCity.dat.dat.gz' after downloading 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz': The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.

I currently do not have the piwik knowledge and time to investigate further but maybe this give the right person a hint?

@jonroberts-anr
Copy link

jonroberts-anr commented May 14, 2017

I blame Cloudflare. On our server we tried to wget and curl the url manually.
wget gives me this:
Connecting to geolite.maxmind.com (geolite.maxmind.com)|104.16.38.47|:80... connected.
HTTP request sent, awaiting response... 503 Service Temporarily Unavailable
2017-05-14 00:09:56 ERROR 503: Service Temporarily Unavailable.

curl gives me this:

<title>Just a moment...</title> <style type="text/css"> html, body {width: 100%; height: 100%; margin: 0; padding: 0;} body {background-color: #ffffff; font-family: Helvetica, Arial, sans-serif; font-size: 100%;} h1 {font-size: 1.5em; color: #404040; text-align: center;} p {font-size: 1em; color: #404040; text-align: center; margin: 10px 0 0 0;} #spinner {margin: 0 auto 30px auto; display: block;} .attribution {margin-top: 20px;} @-webkit-keyframes bubbles { 33%: { -webkit-transform: translateY(10px); transform: translateY(10px); } 66% { -webkit-transform: translateY(-10px); transform: translateY(-10px); } 100% { -webkit-transform: translateY(0); transform: translateY(0); } } @Keyframes bubbles { 33%: { -webkit-transform: translateY(10px); transform: translateY(10px); } 66% { -webkit-transform: translateY(-10px); transform: translateY(-10px); } 100% { -webkit-transform: translateY(0); transform: translateY(0); } } .bubbles { background-color: #404040; width:15px; height: 15px; margin:2px; border-radius:100%; -webkit-animation:bubbles 0.6s 0.07s infinite ease-in-out; animation:bubbles 0.6s 0.07s infinite ease-in-out; -webkit-animation-fill-mode:both; animation-fill-mode:both; display:inline-block; } </style>
<script type="text/javascript">

//x"; t = t.firstChild.href;r = t.match(/https?:\/\//)[0]; t = t.substr(r.length); t = t.substr(0,t.length-1); a = document.getElementById('jschl-answer'); f = document.getElementById('challenge-form'); ;YCbBnNK.D-=+((!+[]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]));YCbBnNK.D+=+((+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));YCbBnNK.D*=!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![];a.value = parseInt(YCbBnNK.D, 10) + t.length; '; 121' f.submit(); }, 4000); }, false); })(); //]]>
</script>

Please turn JavaScript on and reload the page.

<div>
  <div class="bubbles"></div>
  <div class="bubbles"></div>
  <div class="bubbles"></div>
</div>
<h1><span data-translate="checking_browser">Checking your browser before accessing</span> maxmind.com.</h1>

<p data-translate="process_is_automatic">This process is automatic. Your browser will redirect to your requested content shortly.</p>
<p data-translate="allow_5_secs">Please allow up to 5 seconds&hellip;</p>
      <div class="attribution">
        <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=iuam" target="_blank" style="font-size: 12px;">DDoS protection by Cloudflare</a>
        <br>
        Ray ID: 35e99544e3061049
      </div>
  </td>
 
</tr>

@robertoschwald
Copy link

Any news on this? Facing this problem since a long time

@JonTheWong
Copy link

I've been recently getting the same error.
My current error output is for GeoLite2 but I've tried legacy with no success. The URL is giving me a 404 on legacy. Geo2 doesn't work because it doesn't contain a .dat file.

System Check
"Huzzah! There are no problems with your Matomo setup. Give yourself a pat on the back."

Matomo 3.7.0 Current.

ERROR [2019-01-13 11:49:21] 6701  /home/user/public_html/analytics.example.com/plugins/UserCountry/GeoIPAutoUpdater.php(167): GeoIPAutoUpdater: failed to unzip '/home/user/public_html/analytics.example.com/misc/GeoIPCity.dat.tar.gz' after downloading 'https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz': Cannot find GeoIPCity.dat file in tar archive '/home/user/public_html/analytics.example.com/misc/GeoIPCity.dat.tar.gz'!
#0 /home/user/public_html/analytics.example.com/plugins/UserCountry/GeoIPAutoUpdater.php(102): Piwik\Plugins\UserCountry\GeoIPAutoUpdater->downloadFile('loc', 'https://geolite...')
#1 [internal function]: Piwik\Plugins\UserCountry\GeoIPAutoUpdater->update(NULL)
#2 /home/user/public_html/analytics.example.com/core/Scheduler/Scheduler.php(246): call_user_func(Array, NULL)
#3 /home/user/public_html/analytics.example.com/core/Scheduler/Scheduler.php(136): Piwik\Scheduler\Scheduler->executeTask(Object(Piwik\Plugins\UserCountry\GeoIPAutoUpdater))
#4 /home/user/public_html/analytics.example.com/plugins/CoreAdminHome/API.php(58): Piwik\Scheduler\Scheduler->run()
#5 /home/user/public_html/analytics.example.com/core/CronArchive.php(612): Piwik\Plugins\CoreAdminHome\API->runScheduledTasks()
#6 /home/user/public_html/analytics.example.com/core/CronArchive.php(317): Piwik\CronArchive->runScheduledTasks()
#7 /home/user/public_html/analytics.example.com/core/Access.php(626): Piwik\CronArchive->Piwik\{closure}()
#8 /home/user/public_html/analytics.example.com/core/CronArchive.php(319): Piwik\Access::doAsSuperUser(Object(Closure))
#9 /home/user/public_html/analytics.example.com/plugins/CoreConsole/Commands/CoreArchiver.php(27): Piwik\CronArchive->main()
#10 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\Plugins\CoreConsole\Commands\CoreArchiver->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application->doRunCommand(Object(Piwik\Plugins\CoreConsole\Commands\CoreArchiver), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 [internal function]: Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /home/user/public_html/analytics.example.com/core/Console.php(90): call_user_func(Array, Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /home/user/public_html/analytics.example.com/core/Access.php(626): Piwik\Console->Piwik\{closure}()
#16 /home/user/public_html/analytics.example.com/core/Console.php(91): Piwik\Access::doAsSuperUser(Object(Closure))
#17 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(126): Piwik\Console->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /home/user/public_html/analytics.example.com/console(27): Symfony\Component\Console\Application->run()
#19 {main}

Found the original log.

ERROR [2019-01-09 11:45:50] 12232  /home/user/public_html/analytics.example.com/plugins/UserCountry/GeoIPAutoUpdater.php(167): GeoIPAutoUpdater: failed to unzip '/home/user/public_html/analytics.example.com/misc/GeoIPCity.dat.dat.gz' after downloading 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz': The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.
#0 /home/user/public_html/analytics.example.com/plugins/UserCountry/GeoIPAutoUpdater.php(102): Piwik\Plugins\UserCountry\GeoIPAutoUpdater->downloadFile('loc', 'http://geolite....')
#1 [internal function]: Piwik\Plugins\UserCountry\GeoIPAutoUpdater->update(NULL)
#2 /home/user/public_html/analytics.example.com/core/Scheduler/Scheduler.php(246): call_user_func(Array, NULL)
#3 /home/user/public_html/analytics.example.com/core/Scheduler/Scheduler.php(136): Piwik\Scheduler\Scheduler->executeTask(Object(Piwik\Plugins\UserCountry\GeoIPAutoUpdater))
#4 /home/user/public_html/analytics.example.com/plugins/CoreAdminHome/API.php(58): Piwik\Scheduler\Scheduler->run()
#5 /home/user/public_html/analytics.example.com/core/CronArchive.php(612): Piwik\Plugins\CoreAdminHome\API->runScheduledTasks()
#6 /home/user/public_html/analytics.example.com/core/CronArchive.php(317): Piwik\CronArchive->runScheduledTasks()
#7 /home/user/public_html/analytics.example.com/core/Access.php(626): Piwik\CronArchive->Piwik\{closure}()
#8 /home/user/public_html/analytics.example.com/core/CronArchive.php(319): Piwik\Access::doAsSuperUser(Object(Closure))
#9 /home/user/public_html/analytics.example.com/plugins/CoreConsole/Commands/CoreArchiver.php(27): Piwik\CronArchive->main()
#10 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\Plugins\CoreConsole\Commands\CoreArchiver->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application->doRunCommand(Object(Piwik\Plugins\CoreConsole\Commands\CoreArchiver), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 [internal function]: Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /home/user/public_html/analytics.example.com/core/Console.php(90): call_user_func(Array, Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /home/user/public_html/analytics.example.com/core/Access.php(626): Piwik\Console->Piwik\{closure}()
#16 /home/user/public_html/analytics.example.com/core/Console.php(91): Piwik\Access::doAsSuperUser(Object(Closure))
#17 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(126): Piwik\Console->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /home/user/public_html/analytics.example.com/console(27): Symfony\Component\Console\Application->run()
#19 {main}

@OliverTempel
Copy link

Same Error Here.

@OliverTempel
Copy link

MaxMind has changed downloads.
New Files has MaxMind Database Format, so no .dat file is included in download.

@sgiehl
Copy link
Member

sgiehl commented Jan 14, 2019

GeoIP Legacy Lite has been announced to be discontinued about a year ago. Since two weeks there are no downloads available anymore. If you wish to use a free database you need to switch to GeoIP 2. To do this you may need to activate the GeoIp2 plugin in Matomo and configured it in the location provider settings.

will close this issue as we won't/can't fix that anymore

refs #13957

@sgiehl sgiehl closed this as completed Jan 14, 2019
@sgiehl sgiehl added the wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it. label Jan 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it.
Projects
None yet
Development

No branches or pull requests

10 participants