Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GDPR + Add a couple of new platform features especially around validating settings #12558

Merged
merged 80 commits into from Apr 24, 2018

Conversation

tsteur
Copy link
Member

@tsteur tsteur commented Feb 13, 2018

  • GDPR tool to search for visitors (even across all sites)
  • GDPR tool to export such visits
  • GDPR tool to delete all information for such visits
  • GDPR command line tool to anonymize and / or delete information that was tracked in the past.
  • GDPR UI tool to anonymize and/or delete information that was tracked in the past. (cannot really let users select dimensions there)
  • GDPR tracker features to require, give, or remove consent
  • GDPR feature to anonymize userId and orderId
  • GDPR disables newsletter sign up during installation which was enabled by default before.
  • Couple of fixes and improvements
  • New Validators / Settings API:
  • Validators can be used independent of settings API
  • Brings more consistency in error messages and reduces possible validation errors and avoids writing the same validators again and again
  • Validators can define HTML attributes to validate data on client side so we could directly validate the pre-validate the data in the browser (eg required, length, ...). We need to see if the user actually gets informed properly about errors, I think it does not really work and we might have to remove it again
  • Settings can now easily define validations see example.

refs #12600 #12595 #12596 #12598 #12599 #12641

@tsteur tsteur added the Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. label Feb 13, 2018
@@ -158,6 +158,16 @@ public function configureField()

$this->setUiControlIfNeeded($this->config);
$this->checkType($this->config);
if (empty($this->config->availableValues) && in_array($this->type, array(FieldConfig::TYPE_FLOAT, FieldConfig::TYPE_INT, FieldConfig::TYPE_STRING))) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might have to remove this again, or implement the HTML5 validation features into our settings to highlight the errors etc. Will need to be tested with MaterializeCSS etc. If this worked it would be epic.

lang/en.json Outdated
"ValidatorErrorNotANumber": "The value is not a number.",
"ValidatorErrorNumberTooLow": "The value %s is too low. The value needs to be at least %s.",
"ValidatorErrorNumberTooHigh": "The value %s is too high. The value should be at most %s",
"ValidatorErrorCharacterTooShort": "The value contains %s characters but needs to contain at least %s characters.",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to change the messages later.

@@ -140,6 +140,12 @@
}
});

$scope.$watch('inlineHelp', function (val, oldVal) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now makes sure to update inlineHelp when the variable changes.

@@ -122,15 +122,19 @@ var piwikHelper = {
return;
}

angular.element(document).injector().invoke(function($compile) {
var scope = angular.element($element).scope();
angular.element(document).injector().invoke(function($compile, $rootScope) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Options.scope was not working properly and not used. Fixed it and improved the api

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For clarification, does this mean the options.scope object has to be an angular scope object now, rather than just a plain object? If yes, would be help to clarify that in the parameter docs?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that is what it means, I'll update the docs and clarify.

@tsteur tsteur changed the title Add a couple of new platform features especially around validating settings GDPR + Add a couple of new platform features especially around validating settings Mar 12, 2018
<li>Document your use of Matomo within your information asset register.</li>
<li>Make sure that you are inserting our opt-out feature on all pages collecting personal data.</li>
<li>Is your business currently registered to an Privacy Commissioner Office?</li>
<li>Did you clearly informed your visitors on your privacy policy page about their different rights and that you are using Matomo?</li>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by a "Privacy Commissioner Office"? I can't find many (European) sources for this.
Have you asked Ronan if he has an idea on how to write this description?
And I'd write "Have you clearly ..."

@tsteur tsteur added the Needs Review PRs that need a code review label Mar 19, 2018
@tsteur
Copy link
Member Author

tsteur commented Mar 19, 2018

I'll add tests once the general concept for GDPR is reviewed. The content can then be adjusted afterwards.

@tsteur tsteur removed the Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. label Mar 19, 2018
@mattab mattab modified the milestones: 3.3.1, 3.4.0 Mar 19, 2018
@mattab
Copy link
Member

mattab commented Apr 23, 2018

Possible bug report: Anonymised User IDs up to Jan 1st 2017 and got:

  • Status says Done although tooltip is empty?
  • The data seems not anonymised and i'm still seeing User ID visits in Dec 2016

previous log data anon

@mattab
Copy link
Member

mattab commented Apr 23, 2018

In GDPR Tools > Search for a data subject, there is some extra spacing in the dimension picker, on the left of each dimension:

extra spacing

@mattab
Copy link
Member

mattab commented Apr 23, 2018

In the exported data subject's data, let's remote the null columns as to hide implementation details such as number of custom dimensions, or plugins that were installed in the past but maybe not active anymore.

null

@mattab
Copy link
Member

mattab commented Apr 23, 2018

Bug: when exporting data subject's data on "All websites", with an "Admin" user that has only access to one of the websites, getting the error: You can't access this resource as it requires 'view' access for the website id = 3..

@mattab
Copy link
Member

mattab commented Apr 23, 2018

In the test processed XML files, the keys are not translated and appear as the key itself eg. "config_os": "General_Unknown",, but they look localised/translated in the API output for the data subject export. Maybe we could have the tests also show translated strings to reproduce the API behavior?

@tsteur
Copy link
Member Author

tsteur commented Apr 23, 2018

Fixed selector style and job status.

  • All Websites Admin permission error: Couldn't reproduce but can see how this may happen. should be fixed
  • Translation keys: They are translated for me in the real export... and having the translation keys in the tests is fine by me.
  • From export the nulls removed

Will update UI tests later

@mattab mattab merged commit 34c4a85 into 3.x-dev Apr 24, 2018
@mattab mattab deleted the platformfeatures branch April 24, 2018 01:26
InfinityVoid pushed a commit to InfinityVoid/matomo that referenced this pull request Oct 11, 2018
…ting settings (matomo-org#12558)

* add a couple of new platform features especially around validating settings

* more validators and other tweaks

* few tweaks

* allow plugins to add header script in JS tests

* some work on gdpr

* remove email feature

* menu update

* add date tooltip

* make sure privacy has its own icon

* added some text

* add opt out info

* better titles

* add some information about webserver logs

* add log import notice

* give angular some time to render

* show border around iframe

* make sure to place a notification in modal dialog if needed

* when opening a modal, make sure the top part is visible

* disable newsletter sign up by default

* command line tool to anonymize and remove previously tracked raw data

* implement some basic consent features into piwik.js

* minor tweak

* schedule entries to anonymize data in past

* let users anonymize historical data

* better overview

* better naming

* explain consent tracking

* add legitimate interest

* new menu structure for privacy

* use a proper db table for scheduled anonymizations

* add some tracker tests

* some tests and fixes

* couple of fixes and tests

* starting to add some UI tests

* more tweaks

* add missing file

* fix jslint

* have the field full width

* more tweaks

* create directory if needed

* various fixes

* fix some tests

* make sure keys will be always returned in same order

* more fixes

* fix some tests

* apply feedback from review

* fix some tests

* this might fix mysqli tests

* fix some tests

* trying to fix mysqli tests

* fix system test

* fix comment

* remove some duplicated code

* update method name

* trigger event when logs need to be deleted

* add possibility to anonymize userId and orderId

* trigger event in tracking code page

* expire consent in hours

* post another event

* update wording for privacy GDPR

* fix various ui tests

* fix some tests

* trying to fix ui tests

* trigger another test run

* fix system test

* fix some ui tests

* show a table of contents on tracking code page, fix title, add mobile apps and API

* fix ui test

* 3.5.0-b2 update

* Re-generate piwik.js

* fix job status and selector style

* make sure to include only sites with admin access

* some tweaks

* fix some ui tests

* fix some ui tests

* fix test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs Review PRs that need a code review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants