@tsteur opened this Pull Request on February 13th 2018 Member
  • GDPR tool to search for visitors (even across all sites)
  • GDPR tool to export such visits
  • GDPR tool to delete all information for such visits
  • GDPR command line tool to anonymize and / or delete information that was tracked in the past.
  • GDPR UI tool to anonymize and/or delete information that was tracked in the past. (cannot really let users select dimensions there)
  • GDPR tracker features to require, give, or remove consent
  • GDPR feature to anonymize userId and orderId
  • GDPR disables newsletter sign up during installation which was enabled by default before.
  • Couple of fixes and improvements
  • New Validators / Settings API:
    • Validators can be used independent of settings API
    • Brings more consistency in error messages and reduces possible validation errors and avoids writing the same validators again and again
    • Validators can define HTML attributes to validate data on client side so we could directly validate the pre-validate the data in the browser (eg required, length, ...). We need to see if the user actually gets informed properly about errors, I think it does not really work and we might have to remove it again
  • Settings can now easily define validations see example.

refs #12600 #12595 #12596 #12598 #12599 #12641

@tsteur commented on March 19th 2018 Member

I'll add tests once the general concept for GDPR is reviewed. The content can then be adjusted afterwards.

@mattab commented on March 27th 2018 Member

Awesome progress and some great new privacy tools for Matomo!

Feedback:

GDPR tool to search for visitors (even across all sites)
GDPR tool to export such visits
GDPR tool to delete all information for such visits

  • exactly what we needed :+1:
  • about having an audit log of the requests for data access or data deletion, wondering if we need to do it in MVP? In the activity log plugin is good to have, but for all users it seems important to get an audit log for "Requested deletion" and even "Requested access".
  • seems it is using a lot of memory, at least when many visits match, is it maybe something that could be improved? PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in lib s/Zend/Session.php on line 469, referer: /index.php?module=PrivacyManager&action=gdprManageRights&idSite=1&period=day&date=today
  • maybe in core we should add the catch /error screen for memory errors so this error is thrown and shown in the UI (As currently nothing is returned when the 500 error occurs)

GDPR command line tool to anonymize and / or delete information that was tracked in the past.
GDPR UI tool to anonymize and/or delete information that was tracked in the past. (cannot really let users select dimensions there)

  • very useful tools!
  • considering one of the use cases: "I have personal data in my site searches and in Page URLs, how do I anonymise this personal data", then probably one has to select the columns idaction_url and idaction_urlref and (for site searches) idaction_name and possibly idaction_name_ref. i'm wondering what is the easiest solution, maybe we could simply add a couple points in the inline help answering this use case. or adding checkboxes like Unset Page URLs, Unset Page Titles, Unset Site searches, could work but not really needed.

GDPR tracker features to require, give, or remove consent

  • API looks good and clear.
  • How could we handle the case where, new consent is needed and user needs to force all their visitors to re-accept the consents. maybe a feature where they could say "requireConsentIfConsentOlderThan 2018-03-23"
    • if we add such feature, then maybe we could simply remove the optionallyExpireConsentInMs parameter to KISS
  • as agreed, we will work on a full on consent UI later

Regarding how the screens are organised, it's definitely good to have created a new Privacy section...
Not sure how we should organise Privacy Settings / Tools / Manager. Maybe they could be better organised, but didn't think about it too much yet (i'll follow up).

If anyone else has any feedback on this PR, it'll be welcome :+1:

@mattab commented on March 27th 2018 Member

For the raw data anonymisation tool, maybe it would be possible to add the "Visitor id" and "Config id / fingerprint id" in the list of columns, could you explain the difficulties behind these 2 fields?

@tsteur commented on March 27th 2018 Member

They are set to NOT NULL and there is no default value provided. Besides, the idvisitor would also need to be updated in all other log tables etc. Besides obviously a lot of data gets lost etc.

@tsteur commented on March 27th 2018 Member

How could we handle the case where, new consent is needed and user needs to force all their visitors to re-accept the consents. maybe a feature where they could say "requireConsentIfConsentOlderThan 2018-03-23"

The user can get the date the consent was given using a JS tracker method currently which is the most flexible as it lets them implement eg things like "ask for consent again when 6 months have past" etc. We can maybe let users define a callback with requireConsent(function (dateConsentGiven){}) to implement any custom logic and not to restrict to such basic logic but for those things I would first want to have some actual practical use cases when such things are needed and why etc.

@mattab commented on March 27th 2018 Member

The user can get the date the consent was given using a JS tracker method currently which is the most flexible

that is ideal :100: I somehow missed that the method returned the time of the consent... very flexible solution!

@diosmosis commented on April 8th 2018 Member

Left a couple more comments, otherwise looks good to me.

@diosmosis commented on April 16th 2018 Member

Left optional comment on last commit, otherwise looks good.

@mattab commented on April 23rd 2018 Member

Possible bug report: Anonymised User IDs up to Jan 1st 2017 and got:

  • Status says Done although tooltip is empty?
  • The data seems not anonymised and i'm still seeing User ID visits in Dec 2016

previous log data anon

@mattab commented on April 23rd 2018 Member

In GDPR Tools > Search for a data subject, there is some extra spacing in the dimension picker, on the left of each dimension:

extra spacing

@mattab commented on April 23rd 2018 Member

In the exported data subject's data, let's remote the null columns as to hide implementation details such as number of custom dimensions, or plugins that were installed in the past but maybe not active anymore.

null

@mattab commented on April 23rd 2018 Member

Bug: when exporting data subject's data on "All websites", with an "Admin" user that has only access to one of the websites, getting the error: You can't access this resource as it requires 'view' access for the website id = 3..

@mattab commented on April 23rd 2018 Member

In the test processed XML files, the keys are not translated and appear as the key itself eg. "config_os": "General_Unknown",, but they look localised/translated in the API output for the data subject export. Maybe we could have the tests also show translated strings to reproduce the API behavior?

@tsteur commented on April 23rd 2018 Member

Fixed selector style and job status.

  • All Websites Admin permission error: Couldn't reproduce but can see how this may happen. should be fixed
  • Translation keys: They are translated for me in the real export... and having the translation keys in the tests is fine by me.
  • From export the nulls removed

Will update UI tests later

This Pull Request was closed on April 24th 2018
Powered by GitHub Issue Mirror