Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve delete-logs-data memory usage #20038

Open
grzchr15 opened this issue Nov 22, 2022 · 5 comments
Open

Improve delete-logs-data memory usage #20038

grzchr15 opened this issue Nov 22, 2022 · 5 comments
Labels
Bug For errors / faults / flaws / inconsistencies etc. c: Performance For when we could improve the performance / speed of Matomo.

Comments

@grzchr15
Copy link

Expected Behavior

Delete logs data

Current Behavior

Fatal error: Allowed memory size of 1677721600 bytes exhausted (tried to allocate 20480 bytes) in /var/www/clients/client2/web282/web/piwik/plugins/PrivacyManager/Model/DataSubjects.php on line 497

Possible Solution

Steps to Reproduce (for Bugs)

1../console core:delete-logs-data --dates=2011-01-01,2017-12-31 --limit 1000 in /var/www/somepath/web/piwik/core/DataTable/Manager.php
2.PHP Fatal error: Allowed memory size of 1677721600 bytes exhausted (tried to allocate 20480 bytes) in /var/www/clients/client2/web282/web/piwik/plugins/PrivacyManager/Model/DataSubjects.php on line 497

Context

Your Environment

  • Matomo Version: ./console --version
    Matomo version 4.12.3

  • PHP Version:

  • PHP 7.4.33 (cli) (built: Oct 31 2022 10:36:05) ( NTS )
    Copyright (c) The PHP Group
    Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.33, Copyright (c), by Zend Technologies

  • Server Operating System: Redhat 9

  • Additionally installed plugins:

  • ./console plugin:list
    +-----------------------------------+-------------------+---------------+
    | Plugin | Core or optional? | Status |
    +-----------------------------------+-------------------+---------------+
    | Diagnostics | Core | Activated |
    | Resolution | Core | Activated |
    | ScheduledReports | Core | Activated |
    | UserCountryMap | Core | Activated |
    | Live | Core | Activated |
    | PrivacyManager | Core | Activated |
    | ImageGraph | Core | Activated |
    | Annotations | Core | Activated |
    | MobileMessaging | Core | Activated |
    | Login | Core | Activated |
    | SegmentEditor | Core | Activated |
    | Insights | Core | Activated |
    | Morpheus | Core | Activated |
    | Contents | Core | Activated |
    | BulkTracking | Core | Activated |
    | DevicePlugins | Core | Activated |
    | CoreUpdater | Core | Activated |
    | Heartbeat | Core | Activated |
    | Marketplace | Core | Activated |
    | ProfessionalServices | Core | Activated |
    | UserId | Core | Activated |
    | CustomDimensions | Core | Activated |
    | DBStats | Core | Activated |
    | MobileAppMeasurable | Core | Activated |
    | IntranetMeasurable | Core | Activated |
    | TwoFactorAuth | Core | Activated |
    | GeoIp2 | Core | Activated |
    | TagManager | Core | Activated |
    | Tour | Core | Activated |
    | CustomJsTracker | Core | Activated |
    | PagePerformance | Core | Activated |
    | CoreConsole | Core | Activated |
    | Overlay | Core | Activated |
    | Feedback | Core | Activated |
    | LanguagesManager | Core | Activated |
    | CoreAdminHome | Core | Activated |
    | UsersManager | Core | Activated |
    | SitesManager | Core | Activated |
    | Installation | Core | Activated |
    | Monolog | Core | Activated |
    | Intl | Core | Activated |
    | CorePluginsAdmin | Core | Activated |
    | CoreHome | Core | Activated |
    | RssWidget | Core | Activated |
    | CoreVisualizations | Core | Activated |
    | Proxy | Core | Activated |
    | API | Core | Activated |
    | Widgetize | Core | Activated |
    | Transitions | Core | Activated |
    | WebsiteMeasurable | Core | Activated |
    | Actions | Core | Activated |
    | Ecommerce | Core | Activated |
    | VisitorInterest | Core | Activated |
    | VisitTime | Core | Activated |
    | VisitFrequency | Core | Activated |
    | VisitsSummary | Core | Activated |
    | Dashboard | Core | Activated |
    | UserCountry | Core | Activated |
    | Events | Core | Activated |
    | SEO | Core | Activated |
    | CoreVue | Core | Activated |
    | Goals | Core | Activated |
    | MultiSites | Core | Activated |
    | Referrers | Core | Activated |
    | UserLanguage | Core | Activated |
    | DevicesDetection | Core | Activated |
    | TreemapVisualization | Optional | Activated |
    | UsersFlow | Optional | Activated |
    | Migration | Optional | Activated |
    | IPReports | Optional | Activated |
    | CustomVariables | Optional | Activated |
    | WooCommerceAnalytics | Optional | Not activated |
    | LoginSaml | Optional | Activated |
    | Funnels | Optional | Activated |
    | MultiChannelConversionAttribution | Optional | Activated |
    | RollUpReporting | Optional | Activated |
    | LiveTab | Optional | Activated |
    | FormAnalytics | Optional | Activated |
    | ActivityLog | Optional | Activated |
    | MediaAnalytics | Optional | Activated |
    | WhiteLabel | Optional | Activated |
    | HeatmapSessionRecording | Optional | Activated |
    | SecurityInfo | Optional | Activated |
    | CustomReports | Optional | Activated |
    | SearchEngineKeywordsPerformance | Optional | Activated |
    | Bandwidth | Optional | Not activated |
    | LogViewer | Optional | Activated |
    | Provider | Optional | Activated |
    | AbTesting | Optional | Activated |
    +-----------------------------------+-------------------+---------------+

  • Browser:
  • Operating System:
@grzchr15 grzchr15 added Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. To Triage An issue awaiting triage by a Matomo core team member labels Nov 22, 2022
@grzchr15
Copy link
Author

May be linked to any changes happened which cause also #19695

@bx80
Copy link
Contributor

bx80 commented Nov 23, 2022

Hi @grzchr15, thanks for reporting this.

Issue 19695 occurs when archiving data so it's probably not related.

Deleting log data can use quite a bit of memory. The code that ran out of memory in the fatal error above was building a database query with multiple visitor ids. It's possible that this area of code could be made more memory efficient.

Does decreasing the --limit parameter value allow this to complete for you?
Is it possible for you to increase the allowed memory for the process beyond 1.56GiB?

@bx80 bx80 added the Waiting for user feedback Indicates the Matomo team is waiting for feedback from the author or other users. label Nov 23, 2022
@grzchr15
Copy link
Author

retried it with --100 does not change anything

@bx80
Copy link
Contributor

bx80 commented Nov 29, 2022

Thanks for trying that @grzchr15. I'll allocate this issue for prioritization.

Increasing the allowed PHP memory or breaking up the date ranges are probably the best options to get this to run for now.

@bx80 bx80 changed the title Allowed memory size of 1677721600 bytes exhausted at core:delete-logs-data --dates=2011-01-01,2017-12-31 --limit 1000 Improve delete-logs-data memory usage Nov 29, 2022
@bx80 bx80 added this to the For Prioritization milestone Nov 29, 2022
@bx80 bx80 added Bug For errors / faults / flaws / inconsistencies etc. c: Performance For when we could improve the performance / speed of Matomo. and removed Waiting for user feedback Indicates the Matomo team is waiting for feedback from the author or other users. Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. To Triage An issue awaiting triage by a Matomo core team member labels Nov 29, 2022
@grzchr15
Copy link
Author

1.) it would be helpfull to have a way to calculate the needed memory for that.
2.) As running log archive with 4Gbytes works now, i did not try to delete logs for now.
3.) There is cli php memory need and webserver php-fpm needs. Allowing webserver php-fpm to much feels like allowing ddos vectors.
So i hope to be able to setup sensfull resource limits per webrequest and calculate how much memory this instance needs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. c: Performance For when we could improve the performance / speed of Matomo.
Projects
None yet
Development

No branches or pull requests

2 participants