Do I understand it right that if my Piwik server can't use the internet I just have to manually copy the list from https://raw.githubusercontent.com/piwik/referrer-spam-blacklist/master/spammers.txt to /vendor/piwik/referrer-spam-blacklist/spammers.txt and then run ./console scheduled-tasks:run "Piwik\Plugins\CoreAdminHome\Tasks.updateSpammerBlacklist" and then magic will happen ? ;)

Not with this implementation: the auto-updated list is written in the database (and overrides the file spammers.txt).

You could overwrite the file in vendor yes, but that would be overwritten on update (which should be fine since a new Piwik release should have the latest version of the list). You don't have any command to run in that case.

Is it cached or each tracking request selects this list from piwik_option table?

Good point, currently it isn't cached, maybe it should or is it negligible?

Good point, currently it isn't cached, maybe it should or is it negligible?

currently in Tracking API requests we only query log_* tables and not any other table (the round trip to mysql and query runtime are not negligible when you get eg. 500 requests per second.). DB performance would be affected and great tracking API performance is a big part of what makes Piwik scalable. we cache all information from the DB during tracker in the tracker cache.

Ultimately we will need to have performance regression tests to ensure we don't regress performance in Tracking API or other key features of Piwik #7889

+1 to add caching and looks good to merge!

I've added a cache, but I was quite confused by the Tracker\Cache class. I ended up using Piwik\Cache::getLazyCache(). Please advise if there's a better solution.

I guess getEagerCache would be the right choice, as the lazyCache invalidates more often.
@tsteur correct?

I was afraid that the eager cache would load the list (which can be huge, especially in the future) on every single Piwik request/process?

It already does, as the DeviceDetector uses the eager cache. See 2d2b8df

👍 thanks for the link, I'll update to use the eager cache

Looks good 👍 It's awesome to know all piwik users with at least 2.14.0 will have an always up to date spam filter. This will make it much more efficient for all of us to fight referrer spammers.

Anyone reading: feel free to join the fun at: https://github.com/piwik/referrer-spam-blacklist/