@tsteur opened this Issue on December 22nd 2019 Member

see https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

  • This is valid from December 30, 2019 even though it was only announced two days ago (and that during the season where most developers are on holidays)
  • One can't simply download the free GeoIP databases anymore, instead one has to register an account
  • The file is still free of charge
  • But they also use this change to sneakily change the license of the database from the Creative Commons Attribution-ShareAlike to their own end-user license agreement
  • This end-user license agreement is not even available yet, but they are estimating they will publish it on December 23.
  • They reason that they need to collect the user data of all of their users to comply to privacy laws (especially the new Californian one).

  • What does this mean for Matomo:
    The normal GeoIP setup will stop working for new users in a week. For existing Matomo users it will continue to work, but will get out of data.
  • What they don't mention: They can't relicense the existing files, so we can continue to distribute the last database before the change (even though it will become out of date and distributing it might be a challenge)

I reckon as a first step we could temporarily host the latest version of the DB on builds.matomo.org so at least existing downloads won't fail?

Then start implementing an alternative asap? Like eg https://db-ip.com/db/download/ip-to-city-lite
Are there any other suggestions?

@mattab commented on December 22nd 2019 Member

Other suggestions:

  • write a blog post explaining: the situation and what it means for Matomo users (geolocation will get less and less accurate over time if they don't upgrade Matomo), whether people need to do anything, what we're doing to address it, when will it be addressed roughly
  • look into db-ip.com but also this other DB provider: https://lite.ip2location.com - the company and people behind the service have actually built the Matomo plugin themselves https://plugins.matomo.org/IP2Location
  • Shall we release 3.13.1 with the updated URLs very soon then?
@tsteur commented on December 22nd 2019 Member

I suppose we don't want to host the DB ourselves to be GDPR compliant etc for the same reason why MaxMind no longer provides the download directly.

@mattab commented on December 23rd 2019 Member

I suppose we don't want to host the DB ourselves to be GDPR compliant etc for the same reason why MaxMind no longer provides the download directly.

I think we wouldn't want to host it too long, but it would be fine to do it for a few weeks or as long as it takes for us to have an alternative solution eg. ip-db or ip2location?

fyi also asked in https://github.com/maxmind/geoipupdate/issues/61 how it would affect the debian package

@diosmosis commented on December 23rd 2019 Member

@mattab IP2Location requires a sign up as well in order to download the DB. db-ip does not require a sign up and db-ip provides mmdb files which are drop in replacements for geoip2 databases (so we shouldn't have to write any new code, just maybe change the names of the providers and default link to db-ip). We need to add some attribution if db-ip is used, not sure where exactly we'd put that.

There is a checkbox that's required before being able to download the db-ip database. It says we agree to the licensing terms. We may need to add something like that to the db downloader? It might be good to reach out to them and clarify our specific use case first.

@sgiehl commented on December 23rd 2019 Member

I assume db-ip might somewhen do the same as maxmind, if there are any laws that require that.
Let's see how easy the account set up at maxmind will be. Maybe there's a way to do that automatically in the background or so :thinking:

But regardless of that, it should be easy to implement to also allow using dp-ip databases

@Findus23 commented on December 23rd 2019 Member

@sgiehl According to https://github.com/maxmind/geoipupdate/issues/61 they are planning some automated access using the credentials, so this could be implemented in Matomo. Still annoying that everyone will have to sign up for it.

@diosmosis commented on December 23rd 2019 Member

Am I wrong in thinking this is just so maxmind can continue tracking downloads for the geolite databases? Ie, if they didn't do any sort of tracking, would downloading the database still be a "sale" under CCPA?

EDIT: I think they mean to communicate to all users of the geolite database that an IP has to be removed, and that an update to the database is necessary. This would make things more complicated, I think, since it would mean matomo would be responsible for honoring those requests? I guess forcing auto-updates would hopefully be enough.

@Findus23 commented on December 23rd 2019 Member

@diosmosis That's something we will only know after they have published their terms of service. Let's see how "free" they will turn out to be.

@oschwald commented on December 23rd 2019

The blog post has been updated with the new license and sign-up details.

@mattab commented on December 24th 2019 Member

Reading the new terms, and seeing this:

Screenshot from 2019-12-24 13-34-34

This is almost unbelievable really, that our users would have to enter these 4 maxmind agreement (the CC license one is fine)... we can't really ask our users to do this, as far as i can see...

db-ip.com seems to be from a french company as noted in https://www.db-ip.com/privacy.php

I don't see how providing a free DB IP -> Location would be a breach of GDPR in any way. So it should be safe and a good long term solution to use db-ip.com.

@diosmosis could you maybe proceed to implement the changes and test using db-ip.com in Matomo by default instead of Geoip?

@matthewv789 commented on January 1st 2020

The change is to comply with a California privacy law that comes into effect January 1, 2020. In theory this law would affect any such provider, though how they choose to respond will vary.

It does seem that to continue being able to download updates through Matomo, Matomo would need to implement a way to store and make use of a MaxMind license key.

An alternative is to provide instructions for users to install and configure the geoipupdate linux utility.

@lourdas commented on January 2nd 2020

Today I created the MaxMind account and altered my bash script to download the city database using the license key that MaxMind provided me. The instructions at https://dev.maxmind.com/geoip/geoipupdate/ are rather simple and it worked fine for me.

@gamanet commented on January 2nd 2020

In my case it was easy, too...

Download the .deb-Package (or something suitable) https://github.com/maxmind/geoipupdate/releases

Install the package, adding AccountID, LicenseKey, EditionsIDs and changing DatabaseDirectory in "/etc/GeoIP.conf" to /var/www/vhosts/xyz.com/misc/ and adding Cronjob...

But still get GeoIP2AutoUpdater-error-messages... Seems that following mysql-statement needs to be executed:

DELETE FROM matomo_option WHERE option_name like 'geoip.%';

what do you mean?

@dogsbody commented on January 3rd 2020

Can I please just check that everything will continue to work while this is being resolved?

Checking the output of console core:archive shows the failed download but also looks like it's happily using the old MaxMind DB for the time being. Is that correct?

I can implement some work arounds (like above) but would rather get on with my day job and wait for an official fix :-)

Thank you

@diosmosis commented on January 3rd 2020 Member

Checking the output of console core:archive shows the failed download but also looks like it's happily using the old MaxMind DB for the time being. Is that correct?

@dogsbody It will use the old DB as long as it present on your server. If you are using the geolite databases, then in 3.13.1 Matomo will switch automatically to db-ip's (db-ip.com) lite databases which do not require submitting information to db-ip.com. If you'd like to continue using maxmind's geolite databases, you'll have to sign up for a maxmind account, then update Matomo's autoupdater with the new link to the lite databases available in your maxmind account.

@m-cameron commented on January 4th 2020

Based on the recent experience, shouldn't we support multiple geolocation vendors at the same time?

Solely depends to one source is highly vulnerable.

@diosmosis commented on January 4th 2020 Member

@m-cameron MaxMind databases will still be supported, in 3.13.1 db-ip databases will be supported in matomo core and there is already a plugin for ip2location's database on the marketplace. The changes that are being made are for the default database Matomo will recommend using and to seamlessly fix the issue for users currently using maxmind's free database.

@mskala commented on January 4th 2020

Per their FAQ page at https://db-ip.com/faq.php , the free DB-IP city database does not include IPv6 addresses. That may be an unwelcome surprise for someone formerly using the free Maxmind database if a Matomo upgrade automatically switches them to DB-IP.

@HLFH commented on January 4th 2020

Maxmind with geoipupdate works flawlessly. You only have to remove the code related to SetupAutomaticUpdatesOfGeoIP as geoipupdate manages it well. And change DatabaseDirectory in GeoIP.conf to the misc folder of Matomo. It has been quite simple. Only need to document the new Maxmind with geoipupdate process for Matomo.

@diosmosis commented on January 4th 2020 Member

@mskala That is unfortunate and good to know, thanks.

@wolandtel commented on January 5th 2020

MaxMind's instructions aren't clear. So, here're details for URL generation:
URL is: https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&account_id=YOUR_ACCOUNT_ID&license_key=YOUR_LICENCE_KEY&suffix=tar.gz
To obtain account id and license key go to maxmind's account settings Services → My License Key and generate new one.

@neufeind commented on January 5th 2020

For IPv6 then the other option would be ip2location, for which a separate plugin exists and which have a lite-database for ipv4 as well as ipv6. It's just not part of the core. (Does somebody have details about accuracy?) So besides having a solution for MaxMind or for a MaxMind-compatible database (taking into account the file-format) maybe also mentioning the other plugin as one potential solution might be an alternative? If I remember correctly core only mentions installing the geoip2-plugin, right?
https://plugins.matomo.org/Ip2location
https://lite.ip2location.com/database/ip-country-region-city

@diosmosis commented on January 5th 2020 Member

@wolandtel thanks, we'll be putting this information in a faq eventually.

@neufeind ip2location requires a signup which we'd like to avoid by default (we don't want to force our users to provide information for something basic like geolocation). Mostly we want to transition/guide existing users who may not be very technical or or want to provide information past the problem created by maxmind's shift. It seems like for now the choice will be between signing up for a lite product or being ok with not geolocating ipv6. If users want an alternative though, they're entirely free to sign up, we just want to make sure it's not required by default.

@mattab commented on January 8th 2020 Member

We've just released Matomo 3.13.1-rc1 which should fix this issue :+1: You can upgrade easily to this version, see: https://matomo.org/faq/how-to-update/faq_159/
Let us know if you have any feedback!

@mattab commented on January 8th 2020 Member

Feedback @diosmosis

  • After update, the "Download URL" still points to Maxmind which is not expected? Can we change this to the db-ip.com URL automatically?
    Here is what I see:
    Screenshot from 2020-01-08 21-22-05
@wolandtel commented on January 9th 2020

@mattab it's not a good idea. I've changed download URL to my personal (with credentials) and don't expect it'll be changed to db-ip URL.

@diosmosis commented on January 9th 2020 Member

After update, the "Download URL" still points to Maxmind which is not expected? Can we change this to the db-ip.com URL automatically?

demo used geoip1 and the update only changes the url if it uses the geoip2 lite databases.

it's not a good idea. I've changed download URL to my personal (with credentials) and don't expect it'll be changed to db-ip URL.

@wolandtel no worries here, if it's not the exact URL for the lite database, it won't get changed.

@diosmosis commented on January 9th 2020 Member

@mskala just an fyi, we reached out to dbip and confirmed the free databases do have ipv6 addresses. Their faq is likely out of date.

@mskala commented on January 9th 2020

That's good!

@tassoman commented on January 10th 2020 Contributor

To be GDPR compliant you just need to avoid individual identification.
If 01/01/20 you have 1 visit from Sri Lanka you simply shouldn't collect visit's complete IP address.
If the geo-ip database is fitted only with net-masks there's no worry about compliance.

@mpdude commented on January 14th 2020

What exactly should the URL be so I get the new URL when upgrading?

Asking because I removed the old URL earlier this month to get rid of the error messages while waiting for the fix here.

@diosmosis commented on January 15th 2020 Member

@mpdude you can find the link to the database now used by matomo here: https://db-ip.com/db/download/ip-to-city-lite

Note that it is ok if the URL has the month/year in it, the update should work regardless.

@mattab commented on January 16th 2020 Member

This should be fixed in 3.13.1 (due for release in a couple of hours)

This Issue was closed on January 16th 2020
Powered by GitHub Issue Mirror