Other suggestions:

• write a blog post explaining: the situation and what it means for Matomo users (geolocation will get less and less accurate over time if they don't upgrade Matomo), whether people need to do anything, what we're doing to address it, when will it be addressed roughly
• look into db-ip.com but also this other DB provider: https://lite.ip2location.com - the company and people behind the service have actually built the Matomo plugin themselves https://plugins.matomo.org/IP2Location
• Shall we release 3.13.1 with the updated URLs very soon then?

I suppose we don't want to host the DB ourselves to be GDPR compliant etc for the same reason why MaxMind no longer provides the download directly.

I suppose we don't want to host the DB ourselves to be GDPR compliant etc for the same reason why MaxMind no longer provides the download directly.

I think we wouldn't want to host it too long, but it would be fine to do it for a few weeks or as long as it takes for us to have an alternative solution eg. ip-db or ip2location?

fyi also asked in maxmind/geoipupdate#61 how it would affect the debian package

@mattab IP2Location requires a sign up as well in order to download the DB. db-ip does not require a sign up and db-ip provides mmdb files which are drop in replacements for geoip2 databases (so we shouldn't have to write any new code, just maybe change the names of the providers and default link to db-ip). We need to add some attribution if db-ip is used, not sure where exactly we'd put that.

There is a checkbox that's required before being able to download the db-ip database. It says we agree to the licensing terms. We may need to add something like that to the db downloader? It might be good to reach out to them and clarify our specific use case first.

I assume db-ip might somewhen do the same as maxmind, if there are any laws that require that.
Let's see how easy the account set up at maxmind will be. Maybe there's a way to do that automatically in the background or so 🤔

But regardless of that, it should be easy to implement to also allow using dp-ip databases

@sgiehl According to maxmind/geoipupdate#61 they are planning some automated access using the credentials, so this could be implemented in Matomo. Still annoying that everyone will have to sign up for it.

Am I wrong in thinking this is just so maxmind can continue tracking downloads for the geolite databases? Ie, if they didn't do any sort of tracking, would downloading the database still be a "sale" under CCPA?

EDIT: I think they mean to communicate to all users of the geolite database that an IP has to be removed, and that an update to the database is necessary. This would make things more complicated, I think, since it would mean matomo would be responsible for honoring those requests? I guess forcing auto-updates would hopefully be enough.

@diosmosis That's something we will only know after they have published their terms of service. Let's see how "free" they will turn out to be.

The blog post has been updated with the new license and sign-up details.

Reading the new terms, and seeing this:

This is almost unbelievable really, that our users would have to enter these 4 maxmind agreement (the CC license one is fine)... we can't really ask our users to do this, as far as i can see...

db-ip.com seems to be from a french company as noted in https://www.db-ip.com/privacy.php

I don't see how providing a free DB IP -> Location would be a breach of GDPR in any way. So it should be safe and a good long term solution to use db-ip.com.

@diosmosis could you maybe proceed to implement the changes and test using db-ip.com in Matomo by default instead of Geoip?

The change is to comply with a California privacy law that comes into effect January 1, 2020. In theory this law would affect any such provider, though how they choose to respond will vary.

It does seem that to continue being able to download updates through Matomo, Matomo would need to implement a way to store and make use of a MaxMind license key.

An alternative is to provide instructions for users to install and configure the geoipupdate linux utility.

Today I created the MaxMind account and altered my bash script to download the city database using the license key that MaxMind provided me. The instructions at https://dev.maxmind.com/geoip/geoipupdate/ are rather simple and it worked fine for me.

In my case it was easy, too...

Download the .deb-Package (or something suitable) https://github.com/maxmind/geoipupdate/releases

Install the package, adding AccountID, LicenseKey, EditionsIDs and changing DatabaseDirectory in "/etc/GeoIP.conf" to /var/www/vhosts/xyz.com/misc/ and adding Cronjob...

But still get GeoIP2AutoUpdater-error-messages... Seems that following mysql-statement needs to be executed:

DELETE FROM matomo_option WHERE option_name like 'geoip.%';

what do you mean?

Can I please just check that everything will continue to work while this is being resolved?

Checking the output of console core:archive shows the failed download but also looks like it's happily using the old MaxMind DB for the time being. Is that correct?

I can implement some work arounds (like above) but would rather get on with my day job and wait for an official fix :-)

Thank you

Checking the output of console core:archive shows the failed download but also looks like it's happily using the old MaxMind DB for the time being. Is that correct?

@dogsbody It will use the old DB as long as it present on your server. If you are using the geolite databases, then in 3.13.1 Matomo will switch automatically to db-ip's (db-ip.com) lite databases which do not require submitting information to db-ip.com. If you'd like to continue using maxmind's geolite databases, you'll have to sign up for a maxmind account, then update Matomo's autoupdater with the new link to the lite databases available in your maxmind account.

Based on the recent experience, shouldn't we support multiple geolocation vendors at the same time?

Solely depends to one source is highly vulnerable.

@m-cameron MaxMind databases will still be supported, in 3.13.1 db-ip databases will be supported in matomo core and there is already a plugin for ip2location's database on the marketplace. The changes that are being made are for the default database Matomo will recommend using and to seamlessly fix the issue for users currently using maxmind's free database.

Per their FAQ page at https://db-ip.com/faq.php , the free DB-IP city database does not include IPv6 addresses. That may be an unwelcome surprise for someone formerly using the free Maxmind database if a Matomo upgrade automatically switches them to DB-IP.

Maxmind with geoipupdate works flawlessly. You only have to remove the code related to SetupAutomaticUpdatesOfGeoIP as geoipupdate manages it well. And change DatabaseDirectory in GeoIP.conf to the misc folder of Matomo. It has been quite simple. Only need to document the new Maxmind with geoipupdate process for Matomo.

@mskala That is unfortunate and good to know, thanks.

MaxMind's instructions aren't clear. So, here're details for URL generation:
URL is: https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&account_id=YOUR_ACCOUNT_ID&license_key=YOUR_LICENCE_KEY&suffix=tar.gz
To obtain account id and license key go to maxmind's account settings Services → My License Key and generate new one.

For IPv6 then the other option would be ip2location, for which a separate plugin exists and which have a lite-database for ipv4 as well as ipv6. It's just not part of the core. (Does somebody have details about accuracy?) So besides having a solution for MaxMind or for a MaxMind-compatible database (taking into account the file-format) maybe also mentioning the other plugin as one potential solution might be an alternative? If I remember correctly core only mentions installing the geoip2-plugin, right?
https://plugins.matomo.org/Ip2location
https://lite.ip2location.com/database/ip-country-region-city

@wolandtel thanks, we'll be putting this information in a faq eventually.

@neufeind ip2location requires a signup which we'd like to avoid by default (we don't want to force our users to provide information for something basic like geolocation). Mostly we want to transition/guide existing users who may not be very technical or or want to provide information past the problem created by maxmind's shift. It seems like for now the choice will be between signing up for a lite product or being ok with not geolocating ipv6. If users want an alternative though, they're entirely free to sign up, we just want to make sure it's not required by default.

We've just released Matomo 3.13.1-rc1 which should fix this issue 👍 You can upgrade easily to this version, see: https://matomo.org/faq/how-to-update/faq_159/
Let us know if you have any feedback!

Feedback @diosmosis

• After update, the "Download URL" still points to Maxmind which is not expected? Can we change this to the db-ip.com URL automatically?
  Here is what I see:
  

@mattab it's not a good idea. I've changed download URL to my personal (with credentials) and don't expect it'll be changed to db-ip URL.

After update, the "Download URL" still points to Maxmind which is not expected? Can we change this to the db-ip.com URL automatically?

demo used geoip1 and the update only changes the url if it uses the geoip2 lite databases.

it's not a good idea. I've changed download URL to my personal (with credentials) and don't expect it'll be changed to db-ip URL.

@wolandtel no worries here, if it's not the exact URL for the lite database, it won't get changed.

@mskala just an fyi, we reached out to dbip and confirmed the free databases do have ipv6 addresses. Their faq is likely out of date.

That's good!

To be GDPR compliant you just need to avoid individual identification.
If 01/01/20 you have 1 visit from Sri Lanka you simply shouldn't collect visit's complete IP address.
If the geo-ip database is fitted only with net-masks there's no worry about compliance.

What exactly should the URL be so I get the new URL when upgrading?

Asking because I removed the old URL earlier this month to get rid of the error messages while waiting for the fix here.

@mpdude you can find the link to the database now used by matomo here: https://db-ip.com/db/download/ip-to-city-lite

Note that it is ok if the URL has the month/year in it, the update should work regardless.

This should be fixed in 3.13.1 (due for release in a couple of hours)

Does it store 2 or 3 country code characters please? Thanks!