Set a first-party cookie when user opts out of tracking #15184

katebutler · 2019-11-21T21:00:11Z

Part of #14395 (allowing opt-out to work in Chrome on HTTP sites), also related to #6505

tsteur · 2019-11-24T19:45:30Z

js/piwik.js

@@ -7706,6 +7712,32 @@ if (typeof window.Piwik !== 'object') {
        // initialize the Piwik singleton
        addEventListener(windowAlias, 'beforeunload', beforeUnloadHandler, false);

+        window.addEventListener('message', function(e) {
+            try {
+                var data = JSON.parse(e.data);


haven't tested it yet... do we need to check if the message is for us just to avoid any issues with potentially other messages? like include some 'matomo_optout' in the string and then check if e.data && typeof e.data === 'string' && e.data.indexOf('matomo_optout') >= -1? (not tested that particular code... and would need to make sure we post this string in the opt out)

Have added some namespacing of the message attributes to make it less likely that we'll accidentally pick up messages intended for other scripts. The best practice would be to also check the message origin against the domain we expect to receive it from, but I can't find a way to do this since the tracker and the tracked site could be on any domain.
I don't think I need to explicitly check that e.data is defined, is a string etc as I am catching the exceptions from JSON.parse() in situations like this.

tsteur · 2019-11-24T19:46:03Z

js/piwik.js

+                return;
+            }
+
+            if (typeof data.opted_in == 'undefined') {


btw we have some method for this isDefined(data.opt_in) or so is the name

Sweet, using this throughout now

tsteur · 2019-11-24T19:46:17Z

js/piwik.js

+                return;
+            }
+
+            if (typeof data.loaded != 'undefined') {


same here be good to use our isDefined method for this maybe

tsteur · 2019-11-24T19:47:04Z

js/piwik.js

+             * first-party opt-out cookie.
+             */
+            updateOptOutForm: function() {
+                if (typeof window.postMessage === 'undefined') {


here as well be good to use isDefined method

tsteur · 2019-11-24T19:48:19Z

js/piwik.js

+                    if (numAttempts > 1200) {
+                        clearInterval(optOutTimer);
+                    }
+                }, 100);


As DOM interactions are quite expensive in terms of performance, be good to check this maybe only about every second or so? Even every 2 seconds might be fine

The problem is that the optout iframe might be displaying incorrectly to the user (e.g. saying they haven't opted out when they have) until it receives this message. The first time through the loop is usually unsuccessful because the iframe hasn't finished loading yet, and increasing the delay even to 200ms means there is a noticeable pause before the optout iframe updates itself. Maybe we need a new approach - calling postMessage on the iframe before it finishes loading is also causing an error message to be printed to the console which isn't ideal.

Could use setTimeout instead... and each time use something like interval = numAttempts * 100 or * 200 or

tsteur · 2019-11-24T19:50:07Z

plugins/CoreAdminHome/javascripts/optOut.js

@@ -1,22 +1,73 @@
 function submitForm(e, form) {
-    if (e.preventDefault) { // IE8 and below do not support preventDefault


I haven't tested the feature yet but would have expected the preventDefault code and also below the setInterval is still needed?

This code was to prevent the checkbox state from toggling until after the API call had been made, and then refresh the iframe to show the new checkbox state. It seems a bit unnecessary when we are working with the first-party cookies, but if we do want to retain that workflow (i.e. sync the iframe form checkbox with the real world optout value after every change, to make sure that it reflects what will actually happen) I could put back the preventDefault and then fire a message back to the iframe after the first-party cookie has been set in the tracker JS.

…kie; change namespacing of message from mtm_ to maq_; rework optout form initialisation logic to be driven by the optout.js instead of the piwik.js

tsteur · 2019-12-07T04:54:59Z

@Findus23 @MichaelHeerklotz any chance you could give this PR a test as well?

tsteur · 2019-12-10T01:32:38Z

js/piwik.js

+            var tracker = Piwik.getTracker();
+
+            var originHost = getHostName(e.origin);
+            var matomoHost = getHostName(tracker.getPiwikUrl());


Would maybe need to adjust getHostName to check if url is set... otherwise if the tracking code is embedded on the page, but no async tracker is created (which some users do), then it might not return a string? Not sure though... looking at the code it seems getPiwikUrl always returns a string so it is likely fine

tsteur · 2019-12-10T01:32:48Z

I was just testing the opt out. I was embedding the opt out into an html page and opted out. The third party ignore cookie was set. Then I added the tracking code to the same site. I then reloaded the page and it was suddenly saying

You are not opted out

Even though it should say

You are opted out

The problem is that it seems to only check for the first party cookie but not for the third party cookie status here:

This will be pretty much the behaviour for all Matomo sites as none of the visitors that previously unsubscribed would have the first party ignore cookie yet. So it is important to only show "You are not opted out" if neither the first nor the third party ignore cookie is set. If either of them is set, we assume the user is opted out.

I then opted out, and it was saying "opted out" after I clicked on it. However, when I reloaded the page, it was saying again "not opted out".

We should double check that the user was actually opted out before changing the status of the checkbox ideally. And if it didn't change the status, maybe we could show a message explaining the user it didn't work and to retry or to contact the website owner? Maybe could even adjust the message if the opt out is running on http? @mattab Any thoughts? It would basically not work when the site is embedded using http and the user does not have the tracking code on the same page or the opt out iframe domain does not match the tracker domain... In other cases either the first or the third party cookie should work... This message would be shown to actual visitors...
I noticed it was actually not changing the status because I had disableCookies in my tracking code (https://matomo.org/faq/general/faq_157/). I wonder if we need to still execute the methods forgetConsentGiven and rememberConsentGiven even when cookies are disabled? @Findus23 @mattab

Stopped testing afterwards.

mattab · 2019-12-11T04:15:39Z

We should double check that the user was actually opted out before changing the status of the checkbox ideally.

that'd be great.

And if it didn't change the status, maybe we could show a message explaining the user it didn't work and to retry or to contact the website owner? Maybe could even adjust the message if the opt out is running on http? @mattab Any thoughts? It would basically not work when the site is embedded using http and the user does not have the tracking code on the same page or the opt out iframe domain does not match the tracker domain...

If it's possible then be great to explain in the error message why it doesn't work eg Opt-out feature is unfortunately not working because this site is not using https and the tracking code cannot be found on this page. Please contact the website administrator for help. or Opt-out feature is unfortunately not working because this opt-out iframe domain $DOMAIN does not match the analytics service domain $DOMAIN2. Please contact the website administrator for help.

I noticed it was actually not changing the status because I had disableCookies in my tracking code (https://matomo.org/faq/general/faq_157/). I wonder if we need to still execute the methods forgetConsentGiven and rememberConsentGiven even when cookies are disabled? @Findus23 @mattab

not sure of the code details, but the idea would be that consent mechanism should be completely independant of cookies and work the same whether or not cookies are enabled in tracker.

…tside

tsteur · 2019-12-22T23:01:03Z

Note: we still need to ignore disableCookies setting when opting out or in... will create a separate PR for this #15309

tsteur · 2019-12-23T01:48:49Z

FYI made a few tweaks...

making sure it works with more browsers as eg not all browsers support startsWith etc.
Showing an error message if a user has popup blocker enabled (which it is by default on chrome) when changing the status
Showing an error message right away if cookies are disabled since it means it will most certainly not work I reckon
If when the popup closes, the optOut using postMessage did not work, then we may show a warning if we are on http. Generally we only show warnings if we are on HTTP as in all other cases things should work.
We can't fully detect if unsubscribe works because when the popup changes the ignore cookie, the cookies in the opt out frame are not updated and as such only a reload would help. However, we cannot really just reload since we don't really know if we will receive a confirmation message of the opt out through postMessage or not...

@diosmosis @Findus23 could you maybe have a look as well?

diosmosis · 2019-12-24T07:58:24Z

plugins/CoreAdminHome/javascripts/optOut.js

+        matches = e.exec(url);
+
+    return matches ? matches[1] : url;
+}


Would using:

document.createElement('a'); a.href = url; return a.hostname;

be a bit safer here?

diosmosis · 2019-12-24T09:51:27Z

@tsteur looks ok to me though I didn't test locally and it is a fairly complicated PR. I like the interesting use of postMessage here, that seems like it could potentially have other use cases in the tracker (not just opt out and overlay).

It would be nice to be able to unit test the opt out JS if possible (maybe by loading it in karma and testing w/ angular unit tests?). Might be a bit of work to make that possible though; if it's a good idea we could create an issue for now.

…5184)

katebutler added 3 commits November 19, 2019 10:02

First cut at managing opt-out status with a first-party cookie

5c568cc

Send initial state from parent window to iframe once it has loaded

1366b00

Tidying up

71a91c9

tsteur reviewed Nov 24, 2019

View reviewed changes

js/piwik.js Outdated

return;

}

if (typeof data.loaded != 'undefined') {

Copy link

Member

tsteur Nov 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here be good to use our isDefined method for this maybe

tsteur reviewed Nov 24, 2019

View reviewed changes

katebutler added 8 commits November 28, 2019 08:27

PR changes

c33c0ad

Don't send message to iframes other than the intended target

55b5192

Check that messages came from expected origin before changing the coo…

4040bb7

…kie; change namespacing of message from mtm_ to maq_; rework optout form initialisation logic to be driven by the optout.js instead of the piwik.js

Initial JS test for first-party optout cookie

91fd341

Push broken unit tests for review

0fe7416

Final JS test for optout cookie

8dc76f1

Check cookie before sending tracking requests

e60e686

Fix refreshConsentStatus() and add tests

bba7d8b

tsteur reviewed Dec 10, 2019

View reviewed changes

tsteur added this to the 3.13.1 milestone Dec 12, 2019

katebutler added the Needs Review PRs that need a code review label Dec 12, 2019

katebutler added 5 commits December 16, 2019 09:30

Initialize configHasConsent with value of third-party optout cookie

0db3aee

Merge remote-tracking branch 'origin/3.x-dev' into 14395-optout-clien…

a70f220

…tside

Check for potential error conditions in the optout iframe

e2ff848

Fix JSLint errors

c27d964

I18n for optout iframe error messages

14b1f27

tsteur mentioned this pull request Dec 22, 2019

Ignore disableCookie setting when giving consent / removing consent #15309

Merged

tsteur added 3 commits December 23, 2019 12:52

starting to add some fixes and tests

2582396

some tweaks backing up code

e5c7891

more fixes & improvements

f3c5e6c

make sure dom is loaded when checking elements

3e8f58d

diosmosis reviewed Dec 24, 2019

View reviewed changes

tsteur added 3 commits December 31, 2019 14:16

Merge branch '3.x-dev' into 14395-optout-clientside

03433c3

update tracker

c2c429d

fix jslint test

71efd7e

tsteur merged commit 8aeaee6 into 3.x-dev Dec 31, 2019

tsteur deleted the 14395-optout-clientside branch December 31, 2019 02:55

Findus23 mentioned this pull request Jan 28, 2020

Matomo can't track IE <9 any more #15477

Closed

Findus23 mentioned this pull request Feb 28, 2020

Using Opt-Out iFrame twice doesn't work #15629

Closed

jonasgrilleres pushed a commit to 1024pix/pix-analytics that referenced this pull request Sep 22, 2020

Set a first-party cookie when user opts out of tracking (matomo-org#1…

0158300

…5184)

jbuget pushed a commit to 1024pix/pix-analytics that referenced this pull request Sep 26, 2020

Set a first-party cookie when user opts out of tracking (matomo-org#1…

8119dc2

…5184)

tsteur mentioned this pull request Nov 2, 2020

Optout function in an iframe on iOS/MacOS no longer works #16648

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set a first-party cookie when user opts out of tracking #15184

Set a first-party cookie when user opts out of tracking #15184

katebutler commented Nov 21, 2019

tsteur Nov 24, 2019

katebutler Nov 27, 2019

tsteur Nov 24, 2019

katebutler Nov 27, 2019

tsteur Nov 24, 2019

tsteur Nov 24, 2019

tsteur Nov 24, 2019 •

edited

katebutler Nov 27, 2019

tsteur Nov 27, 2019 •

edited

tsteur Nov 24, 2019

katebutler Nov 27, 2019

tsteur commented Dec 7, 2019

tsteur Dec 10, 2019

tsteur commented Dec 10, 2019

mattab commented Dec 11, 2019

tsteur commented Dec 22, 2019 •

edited

tsteur commented Dec 23, 2019

diosmosis Dec 24, 2019

diosmosis commented Dec 24, 2019

		@@ -1,22 +1,73 @@
		function submitForm(e, form) {
		if (e.preventDefault) { // IE8 and below do not support preventDefault

Set a first-party cookie when user opts out of tracking #15184

Set a first-party cookie when user opts out of tracking #15184

Conversation

katebutler commented Nov 21, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsteur Nov 24, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsteur Nov 27, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsteur commented Dec 7, 2019

Choose a reason for hiding this comment

tsteur commented Dec 10, 2019

mattab commented Dec 11, 2019

tsteur commented Dec 22, 2019 • edited

tsteur commented Dec 23, 2019

Choose a reason for hiding this comment

diosmosis commented Dec 24, 2019

tsteur Nov 24, 2019 •

edited

tsteur Nov 27, 2019 •

edited

tsteur commented Dec 22, 2019 •

edited