Allow using and auto-updating dbip databases and default to db-ip.com lite in the UI #15319

diosmosis · 2019-12-27T17:34:22Z

…RL is used and download DB IP database to special file so users will know they are not using a maxmind db.

…ite.

sgiehl · 2019-12-28T11:19:04Z

@mattab could you maybe get in touch with DB-IP, to get access to their paid databases? maybe also the one including ISP data? Would be good if we could test those as well.

plugins/GeoIp2/LocationProvider/GeoIp2.php

tsteur · 2019-12-29T19:44:25Z

BTW haven't looked at the code but do we want to auto update the URL for people using the free maxmind DB?

sgiehl · 2019-12-29T19:55:05Z

guess it would make sense to do that in an update script

plugins/GeoIp2/GeoIP2AutoUpdater.php

mattab · 2019-12-29T20:50:12Z

could you maybe get in touch with DB-IP, to get access to their paid databases? maybe also the one including ISP data? Would be good if we could test those as well.

Just requested to db-ip team if they're willing to send us a commercial DB set, will let you know when we hear back. Btw i noticed the commercial DBs are quite large, see:

Would this still work fine even though the DB can be up to 4DB large?

diosmosis · 2019-12-29T20:55:05Z

Looks like those are for the CSV files, the mmdb files are smaller. Though they still seem to be quite large (~350mb for city db). The download might take a while but for once a month it should be ok?

EDIT: I'm also not sure if that's gzipped or not.

diosmosis · 2019-12-29T21:24:34Z

Updated the PR

plugins/GeoIp2/GeoIP2AutoUpdater.php

plugins/GeoIp2/LocationProvider/GeoIp2.php

core/Updates/3.13.1-b2.php

mattab · 2020-01-01T03:37:03Z

FYI @diosmosis our affiliate link for db-ip.com is https://db-ip.com/?refid=mtm or especially appending ?refid=mtm to all links to db-ip.com. With maxmind we had a similar affiliate link. Can you check and add the parameter to all links within the app to db-ip.com?

…pe of database is going to be downloaded, add support for other db-ip DBs, use enterprise geoip db for ISP detection if present, add db-ip referrer query param.

diosmosis · 2020-01-02T21:12:48Z

@sgiehl @tsteur updated to work w/ purchased dbip urls. tested w/ all of them and tested w/ geolite2 city just to make sure it still works.

tsteur · 2020-01-02T22:32:32Z

BTW: Not sure if important... once it failed for me and then I got this notice:

Notice: Undefined index: region_code in plugins/GeoIp2/LocationProvider/GeoIp2.php on line 101

diosmosis · 2020-01-02T23:46:46Z

@tsteur should be fixed, let me know if you still get it

plugins/GeoIp2/LocationProvider/GeoIp2.php

plugins/GeoIp2/LocationProvider/GeoIp2/Php.php

sgiehl · 2020-01-03T18:52:05Z

plugins/GeoIp2/LocationProvider/GeoIp2.php

    {
        static $result = null;
        if (is_null($result)) {
            $expected = array(self::COUNTRY_CODE_KEY => 'FR',
-                self::REGION_CODE_KEY  => 'BFC',
+                self::REGION_CODE_KEY => 'BFC',


The DB-IP Lite database returns no region for this IP, so the provider is reported as broken.
In relation to #15317 we maybe shouldn't use only one IP to check if the results are ok. Maybe we could have one IP per continent and go through the list until one matches exactly? Any idea where to get some IPs with static results?

Or we might use different ips/results depending on the provider / database type

What about using the test database we include for system tests? Could we load that so we can control the IP/location? I started on supporting multiple ip addresses, but we'd have to have quite a few and they might change...

I guess that wouldn't work for server modules... though it looks like servermodule overrides isWorking(). If we move that logic to the Php class it might work I think.

I think I'm going to make the check less strict and just check that there's at least some data in the provider. Users can always see their own IP geolocated to check for accuracy.

…her priority files so the new downloaded one will be used, and rename some translations.

mattab · 2020-01-08T01:55:41Z

@diosmosis @sgiehl As soon as this is merged we'll release another beta for testing, looking forward to getting it merged today or tomorrow 🚀

… lite in the UI (matomo-org#15319)

sgiehl and others added 3 commits December 24, 2019 11:03

Allow using Db-IP GeoIP2 databases

29383e0

Update db-ip date to today in URL so autoupdater will work when new U…

a2ee0e6

…RL is used and download DB IP database to special file so users will know they are not using a maxmind db.

Add DBIP to UI text and default to DBIP lite database instead of geol…

df77959

…ite.

diosmosis added the Needs Review PRs that need a code review label Dec 27, 2019

diosmosis added this to the 3.13.1 milestone Dec 27, 2019

sgiehl reviewed Dec 29, 2019

View reviewed changes

plugins/GeoIp2/LocationProvider/GeoIp2.php Outdated Show resolved Hide resolved

sgiehl reviewed Dec 29, 2019

View reviewed changes

plugins/GeoIp2/GeoIP2AutoUpdater.php Outdated Show resolved Hide resolved

diosmosis added 3 commits December 29, 2019 13:05

Add update to update geoip auto updater URL.

3357f08

fix warning

7e32443

Update initial geoip configure wizard to use dbip databases.

9da23f7

tsteur reviewed Dec 29, 2019

View reviewed changes

plugins/GeoIp2/GeoIP2AutoUpdater.php Show resolved Hide resolved

tsteur reviewed Dec 29, 2019

View reviewed changes

plugins/GeoIp2/LocationProvider/GeoIp2.php Show resolved Hide resolved

sgiehl reviewed Dec 30, 2019

View reviewed changes

core/Updates/3.13.1-b2.php Show resolved Hide resolved

In update also change geoip2 lite country DB URLs.

a062eb4

mattab changed the title ~~Allow using and auto-updating dbip databases and default to dbip lite in the UI~~ Allow using and auto-updating dbip databases and default to db-ip.copm lite in the UI Jan 1, 2020

mattab changed the title ~~Allow using and auto-updating dbip databases and default to db-ip.copm lite in the UI~~ Allow using and auto-updating dbip databases and default to db-ip.com lite in the UI Jan 1, 2020

sgiehl mentioned this pull request Jan 1, 2020

Geoip2 update: Suggested server cannot be resolved #15339

Closed

Support db-ip purchased URLs for whom there is no way to tell what ty…

4f7020d

…pe of database is going to be downloaded, add support for other db-ip DBs, use enterprise geoip db for ISP detection if present, add db-ip referrer query param.

Fix notice

adddb49

sgiehl reviewed Jan 3, 2020

View reviewed changes

plugins/GeoIp2/LocationProvider/GeoIp2.php Outdated Show resolved Hide resolved

sgiehl reviewed Jan 3, 2020

View reviewed changes

plugins/GeoIp2/LocationProvider/GeoIp2/Php.php Show resolved Hide resolved

sgiehl reviewed Jan 3, 2020

View reviewed changes

diosmosis added 5 commits January 5, 2020 14:02

Merge branch '3.x-dev' into dbip

2a59c15

Apply some review feedback.

b85537b

Merge branch '3.x-dev' into dbip

afba8b9

Simplify geoip2 isWorking functionality, when autoupdating rename hig…

fe0d10b

…her priority files so the new downloaded one will be used, and rename some translations.

fix tests

709f38d

tsteur merged commit 4ad9f4a into 3.x-dev Jan 8, 2020

tsteur deleted the dbip branch January 8, 2020 02:30

mattab added the Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. label Jan 16, 2020

jonasgrilleres pushed a commit to 1024pix/pix-analytics that referenced this pull request Sep 22, 2020

Allow using and auto-updating dbip databases and default to db-ip.com…

00c88a6

… lite in the UI (matomo-org#15319)

jbuget pushed a commit to 1024pix/pix-analytics that referenced this pull request Sep 26, 2020

Allow using and auto-updating dbip databases and default to db-ip.com…

2d9c13b

… lite in the UI (matomo-org#15319)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow using and auto-updating dbip databases and default to db-ip.com lite in the UI #15319

Allow using and auto-updating dbip databases and default to db-ip.com lite in the UI #15319

diosmosis commented Dec 27, 2019 •

edited by mattab

sgiehl commented Dec 28, 2019

tsteur commented Dec 29, 2019

sgiehl commented Dec 29, 2019

mattab commented Dec 29, 2019

diosmosis commented Dec 29, 2019 •

edited

diosmosis commented Dec 29, 2019

mattab commented Jan 1, 2020

diosmosis commented Jan 2, 2020

tsteur commented Jan 2, 2020

diosmosis commented Jan 2, 2020

sgiehl Jan 3, 2020

sgiehl Jan 3, 2020 •

edited

diosmosis Jan 5, 2020

diosmosis Jan 5, 2020

diosmosis Jan 7, 2020

mattab commented Jan 8, 2020

Allow using and auto-updating dbip databases and default to db-ip.com lite in the UI #15319

Allow using and auto-updating dbip databases and default to db-ip.com lite in the UI #15319

Conversation

diosmosis commented Dec 27, 2019 • edited by mattab

sgiehl commented Dec 28, 2019

tsteur commented Dec 29, 2019

sgiehl commented Dec 29, 2019

mattab commented Dec 29, 2019

diosmosis commented Dec 29, 2019 • edited

diosmosis commented Dec 29, 2019

mattab commented Jan 1, 2020

diosmosis commented Jan 2, 2020

tsteur commented Jan 2, 2020

diosmosis commented Jan 2, 2020

sgiehl Jan 3, 2020

Choose a reason for hiding this comment

sgiehl Jan 3, 2020 • edited

Choose a reason for hiding this comment

diosmosis Jan 5, 2020

Choose a reason for hiding this comment

diosmosis Jan 5, 2020

Choose a reason for hiding this comment

diosmosis Jan 7, 2020

Choose a reason for hiding this comment

mattab commented Jan 8, 2020

diosmosis commented Dec 27, 2019 •

edited by mattab

diosmosis commented Dec 29, 2019 •

edited

sgiehl Jan 3, 2020 •

edited