Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Built-In Country Pre-Filter #16352

Closed
dev-101 opened this issue Aug 29, 2020 · 20 comments
Closed

Built-In Country Pre-Filter #16352

dev-101 opened this issue Aug 29, 2020 · 20 comments
Labels
c: New plugin For features that probably will not be added to Matomo, but could be implemented as plugins. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. trackingspam For issues related to receiving tracking requests from spammers and bots.

Comments

@dev-101
Copy link

dev-101 commented Aug 29, 2020

Hi, it would be very useful if we could tell Matomo to either:

1. only track visits from specific target countries
2. or completely exclude visitors from specific countries

By exclude or track, I mean log into database or completely ignore / drop them. This would save so much resources, streamline our analytics work and keep database resources low.

Yes, I know we have post-filters, but that's not it. It is not good enough.

Also, yeah, we can use 3rd party scripts, but problem with custom country blockers is that it adds additional layer of complexity on top of Matomo, we have to maintain a script that depends on 3rd party updates etc. and Matomo already has all that built-in. Once it determines a country from a visitor's IP, it can simply decide whether it should store it or drop it.

Is there a way to actually build a simple plugin that can do this for us? It should be a simple hard-coded array of countries, I don't mind. It does not have to work in real-time, say, a cron-based script that can clean-up all entries in database every 24 hours that match certain criteria (e.g. visitor's country field). Similar to what we have in GDPR tools now.

Thanks!

@dev-101 dev-101 changed the title Built-In Country Filter Built-In Country Pre-Filter Aug 29, 2020
@Findus23
Copy link
Member

Findus23 commented Aug 29, 2020

Hi,

I think this might be easily implemented as a plugin that during tracking checks if the detected country match your list and if so, simply drop the request (similar to DNT).

In theory listening to this event

Piwik::postEvent('Tracker.isExcludedVisit', array(&$excluded, $this->request));

should be enough, but I am not sure if at this point Geolocation has already run. If so, it should be pretty easy to implement.

@Findus23 Findus23 added Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. c: New plugin For features that probably will not be added to Matomo, but could be implemented as plugins. labels Aug 29, 2020
@dev-101
Copy link
Author

dev-101 commented Aug 29, 2020

Hi, thanks, I do wish to go that route at first, this is just a suggestion, as I think this actually can be a very useful feature.
I don't know where to start with the plugin, though, I have to really dig into core and see what methods are used to for e.g. remove visits on GDPR page.

add: I see your edited answer. I will try to test it later today. Thanks!

@Findus23
Copy link
Member

It seems like my simple idea using Tracker.isExcludedVisit doesn't work as at that point one only has Request which holds the parameters of the http request. I still think that excluding during the tracker is a better idea than retroactively removing them from the database as that means invalidating and reprocessing a lot of reports.

@sgiehl
Copy link
Member

sgiehl commented Aug 29, 2020

@dev-101 actually that could work using Tracker.isExcluded event. Adding something like this into a config.php in config directory should work:

<?php

return [
    'observers.global' => \DI\add([[
        'Tracker.isExcludedVisit',
            function(&$excluded, \Piwik\Tracker\Request $request) {

                $provider = \Piwik\Plugins\UserCountry\LocationProvider::getProviderById(\Piwik\Common::getCurrentLocationProviderId());

                $location = $provider->getLocation(['ip' => $request->getIp()]);

                if (strtolower($location[\Piwik\Plugins\UserCountry\LocationProvider::COUNTRY_CODE_KEY]) !== 'de') {
                    $excluded = true;
                }

            }
    ]])
];

That should exclude all visits not coming from Germany.

Note: This might not be the optimal solution, as would do the geo location twice per request.

@dev-101
Copy link
Author

dev-101 commented Aug 29, 2020

Thank you, it looks promising, although putting this into config.ini.php will probably get overwritten during the next update (I assume). Wonder if that code can be placed inside a plugin?

I have added the code, but for now I am just simulating it. Thanks!

@dev-101
Copy link
Author

dev-101 commented Aug 29, 2020

Unfortunately, it's not working:

PHP Fatal error: Uncaught Piwik\Ini\IniReadingException: Unable to read INI file {.../config/config.ini.php}: Syntax error in INI configuration: syntax error, unexpected $end, expecting ']' in Unknown on line 4

It seems that we must not put PHP code inside ini file, as it confuses Piwik/Matomo.


update: I tried putting this code inside return array of global.php (seems "logical"), but can't say it is working.

@tsteur tsteur added this to the Backlog (Help wanted) milestone Aug 30, 2020
@Findus23
Copy link
Member

The hint by @sgiehl helped me create a plugin that at least in my limited test seems to work fine:
https://github.com/Findus23/plugin-ExcludeCountries
you can download the zip from https://github.com/Findus23/plugin-ExcludeCountries/archive/main.zip, rename the directory to ExcludeCountries, move it to plugins/, enable it in the plugin view, remove the part from the global.php and check if it is working with https://developer.matomo.org/api-reference/tracking-api#debugging-the-tracker

DEBUG ExcludeCountries[2020-08-30 08:55:31 UTC] [519fe] request excluded by ExcludeCountries plugin (at)
DEBUG CoreHome[2020-08-30 08:55:31 UTC] [519fe] Visitor excluded.

@dev-101
Copy link
Author

dev-101 commented Aug 30, 2020

Thank you, Findus23!
I have installed it and configured, now have to wait & see. Many thanks!

@dev-101
Copy link
Author

dev-101 commented Aug 30, 2020

minor issue:

PHP Notice: Undefined index: languageCode in /ExcludeCountries/ExcludeCountries.php on line 69

But, so far, tracking of included countries seems to be working fine.

@Findus23
Copy link
Member

@dev-101 Not sure why exactly this happens, but I now exclude invalid settings.

@dev-101
Copy link
Author

dev-101 commented Aug 30, 2020

According to time stamps, it was at the time I installed and configured plugin, there were like 20-30 lines of above notice in the log. Afterwards nothing, so it works ok. I "fixed" it with @

@dev-101
Copy link
Author

dev-101 commented Aug 31, 2020

I will close this issue now.
Any bug report re plugin will be posted in its own repo.

Thanks everyone 👍

@dev-101 dev-101 closed this as completed Aug 31, 2020
@tsteur
Copy link
Member

tsteur commented Aug 31, 2020

@dev-101 I'll reopen the issue for now just in case anyone wants to develop something like this as a plugin

@tsteur tsteur reopened this Aug 31, 2020
@dev-101
Copy link
Author

dev-101 commented Aug 31, 2020

Ok :)

btw. Findus23's plugin works great so far... it's a done deal imho. It is fully configurable in the General Settings page, at least, it satisfies both points 1. and 2. from the opening post.

@Findus23
Copy link
Member

Findus23 commented Sep 1, 2020

@Findus23 Findus23 closed this as completed Sep 1, 2020
@tsteur tsteur added the not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. label Sep 9, 2020
@tsteur tsteur added the trackingspam For issues related to receiving tracking requests from spammers and bots. label Nov 25, 2020
@dev-101
Copy link
Author

dev-101 commented Dec 4, 2020

Hi @Findus23 could you please bump plugin compatibility with Matomo 4? Thanks!

edit: I see Matomo pulled updated version automatically after update, I received a warning before upgrade about compatibility and that it will be disabled after upgrade. Sorry.

@Findus23
Copy link
Member

Findus23 commented Dec 4, 2020

Hi @dev-101,

No problem. I wasn't sure myself if I didn't just forget the plugin.

@Findus23
Copy link
Member

@dev-101 Just for your information: There is an official plugin currently in development that supports everything my plugin can do and even more:

https://github.com/matomo-org/plugin-TrackingSpamPrevention

@dev-101
Copy link
Author

dev-101 commented Dec 31, 2020

Thanks, this is great! Happy NY :)

@dev-101
Copy link
Author

dev-101 commented Jul 11, 2021

Just a heads-up if someone is using this plugin (version 0.2.0), it caused tracking issues on my website out-of-the-blue -- last 4 days tracking stopped and here's the report:

Last 4 days one of my Matomo installation (shared hosting, cPanel, Linux, PHP 7.3.8) stopped tracking visitors. GA works fine.

Here's what I found in system reports:

ERROR The configured geolocation provider DBIP / GeoIP 2 (Php) is broken. Please fix the provider or configure another one to get geolocation working again.

matomo-sys-check

matomo-js-500-error

Stack trace:
#0 [internal function]: Piwik\Plugins\ExcludeCountries\ExcludeCountries->checkExcludedCountry(false, Object(Piwik\Tracker\Request))
#1 /matomo/core/EventDispatcher.php(141): call_user_func_array(Array, Array)
#2 /matomo/core/Piwik.php(809): Piwik\EventDispatcher->postEvent('Tracker.isExclu...', Array, false, Array)
#3 /matomo/core/Tracker/VisitExcluded.php(97): Piwik\Piwik::postEvent('Tracker.isExclu...', Array)
#4 /matomo/plugins/CoreHome/Tracker/VisitRequestProcessor.php(100): Piwik\Tracker\VisitExcluded->isExcluded()
#5 /matomo/core/Tracker/Visit.php(164): Piwik\Plugins\CoreHome\Tracker\VisitRequestProcessor->processRequestParams(Object(Piwik\Tracker\Visit\VisitProperties), Object(Piwik\Tracker\Request))
#6 in /matomo/plugins/ExcludeCountries/ExcludeCountries.php on line 73

I haven't changed anything in the past few months besides updating Matomo.

Contrary to what above reports might indicate, the issue is within this plugin, simply deactivating it resolved tracking and other problems right away.

I know there was a work on a new plugin, but I didn't switch to it. Maybe that was the problem...

edit: switched to newer plugin, problem fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: New plugin For features that probably will not be added to Matomo, but could be implemented as plugins. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. trackingspam For issues related to receiving tracking requests from spammers and bots.
Projects
None yet
Development

No branches or pull requests

4 participants