Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can we make transitions to use archived data? #14172

Open
mikkeschiren opened this issue Mar 10, 2019 · 5 comments
Open

Can we make transitions to use archived data? #14172

mikkeschiren opened this issue Mar 10, 2019 · 5 comments
Labels
c: Performance For when we could improve the performance / speed of Matomo.

Comments

@mikkeschiren
Copy link
Contributor

Have not looked so deep into this - but is it possible to make transitions use archived data instead of the logs? If it it is not easily done - what is needed to change to make this possible?

@mikkeschiren
Copy link
Contributor Author

Use case: We have instances wit a lot of logs - and we need to clean up the logs very often, we get all the reports on archived data - but Transitions goes direct to the log tables. And after we clean up old logs, we do not get any data for Transitions.

@tsteur tsteur added the c: Performance For when we could improve the performance / speed of Matomo. label Mar 10, 2019
@tsteur
Copy link
Member

tsteur commented Mar 10, 2019

It's not really planned currently but may be good to do at some point. Especially now that the report is more exposed and could cause a lot more performance issues as it's known to be possibly slow with lots of logs.

@tsteur tsteur added Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. and removed Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. labels Mar 10, 2019
@mikkeschiren
Copy link
Contributor Author

Ok, I will try to look into this in the near feature.

@mfb
Copy link

mfb commented Sep 11, 2019

We'd love to be able to generate Transitions reports from archived records. Our privacy policy requires that we archive/aggregate logs on a weekly basis, but we want to be able to analyze transitions from the past month or so..

@mattab mattab added the Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. label Sep 22, 2019
@mattab
Copy link
Member

mattab commented Oct 9, 2019

We've discussed it internally and we are a bit worried of archiving the Transitions data because it represents a lot of data to aggregate + a lot of slow running SQL queries to get this data daily. For each Page URL and Page title we'd need to store the last 10-30 pages/events/referrers and next 10-30. So that's a lot of string data/url to store for each URL/page title on the site.

Instead we have another idea:
Maybe it would be possible for you to keep old RAW logs for a longer time,
but make sure that the RAW logs you keep in Matomo are fully anonymised.

Maybe we could build an easier feature for "Full anonymisation" of the data, to fully remove any potential personal data, but would still leave the actual pageviews transitions data so we could still process and report on Transitions using RAW data?

This was discussed in #12737 Enable "Super Privacy" mode to not track any personal data, aka "I do not want to be bothered with GDPR"

And also more partially in other issues:

@mattab mattab removed the Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. label Oct 21, 2019
@mattab mattab added this to the Backlog (Help wanted) milestone Jan 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo.
Projects
None yet
Development

No branches or pull requests

4 participants