Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show Warning if Deleting Old Data Does Not Work #18839

Open
mritzmann opened this issue Feb 23, 2022 · 6 comments
Open

Show Warning if Deleting Old Data Does Not Work #18839

mritzmann opened this issue Feb 23, 2022 · 6 comments
Labels
c: Privacy For issues that impact or improve the privacy. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.

Comments

@mritzmann
Copy link

mritzmann commented Feb 23, 2022

I have a Matomo installation which became slower and slower over time. The cause turned out to be that the table matomo_log_visit contained several years of RAW data and thus became several GB in size. This affected the reading of this table noticeably. In the settings, Regularly delete old raw data is set to 30 days, and a cron job is set up.

It turns out: the cron job ran out of memory every time during Tasks.deleteLogData.

Matomo displays a notification in the backend when archiving via cronjob fails. However, when tasks fails, this is not displayed anywhere. My wish would be to proactively display failed tasks and notify the user that the tasks have not been fully completed. The ScheduledReports plugin does not show the error either.

Summary

  • Display notification when Tasks.deleteLogData are not completed successfully for a long time.
  • This is important for performance reasons and maybe for reasons of the General Data Protection Regulation (GDPR).

Your Environment

  • Matomo Version: 4.7.1
  • PHP Version: 7.4.28
  • Server Operating System: Debian 10
@mritzmann mritzmann added the Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. label Feb 23, 2022
@MatomoForumNotifications

This issue has been mentioned on Matomo forums. There might be relevant details there:

https://forum.matomo.org/t/high-traffic-sites-and-custom-period-date-ranges-selection/44717/2

@sgiehl
Copy link
Member

sgiehl commented Feb 23, 2022

@mritzmann Thanks for creating the issue.
Which size is your memory limit set to? Just wondering if you limit is quite small or if the memory consumption of our script is very high in that case.

@mritzmann
Copy link
Author

Hello @sgiehl, thank you very much for your reply.

Which size is your memory limit set to?

The PHP memory limit was set to 4G.

$ cat php.ini | grep memory_limit
memory_limit = 4G

$ php -r 'echo ini_get("memory_limit");'      
4G

@sgiehl sgiehl added the c: Privacy For issues that impact or improve the privacy. label Feb 24, 2022
@sgiehl
Copy link
Member

sgiehl commented Feb 24, 2022

Ok. That should indeed be enough to clean up the data. I guess we need to investigate how our code to remove old visits currently works. Maybe it tries to query all data before removing it or has another memory lack we should close.
ping @tsteur

@tsteur
Copy link
Member

tsteur commented Feb 24, 2022

Had a quick look at the code and the delete log data task should be quite low memory and we delete max 2000 visits at once there and shouldn't save much in memory. On our own instance have never seen any memory issues there.

This may trigger also the log_actions cleanup though. @mritzmann could you maybe add below entry to your config/config.ini.php to see if it still happens? This effectively should delay running the log_action cleanup by a lot of days and basically disables it

[Deletelogs]
delete_logs_unused_actions_schedule_lowest_interval = 9000000

@mritzmann
Copy link
Author

mritzmann commented Mar 10, 2022

I am sorry, but I can no longer reproduce the problem, as I have since deleted several GB of RAW data (to solve the problem described above). The cronjob is currently running successfully for me. Therefore, it probably doesn't make sense for me to check settings like delete_logs_unused_actions_schedule_lowest_interval.

But I still think that some kind of monitoring check (for example a notification in webui) would be interesting. This would be less a bug, but more a feature request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Privacy For issues that impact or improve the privacy. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests

5 participants