@grzchr15 opened this Issue on November 22nd 2022

Expected Behavior

Delete logs data

Current Behavior

Fatal error: Allowed memory size of 1677721600 bytes exhausted (tried to allocate 20480 bytes) in /var/www/clients/client2/web282/web/piwik/plugins/PrivacyManager/Model/DataSubjects.php on line 497

Possible Solution

Steps to Reproduce (for Bugs)

1../console core:delete-logs-data --dates=2011-01-01,2017-12-31 --limit 1000 in /var/www/somepath/web/piwik/core/DataTable/Manager.php
2.PHP Fatal error: Allowed memory size of 1677721600 bytes exhausted (tried to allocate 20480 bytes) in /var/www/clients/client2/web282/web/piwik/plugins/PrivacyManager/Model/DataSubjects.php on line 497

Context

Your Environment

  • Matomo Version: ./console --version
    Matomo version 4.12.3

  • PHP Version:

  • PHP 7.4.33 (cli) (built: Oct 31 2022 10:36:05) ( NTS )
    Copyright (c) The PHP Group
    Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.33, Copyright (c), by Zend Technologies

  • Server Operating System: Redhat 9

  • Additionally installed plugins:

  • ./console plugin:list
    +-----------------------------------+-------------------+---------------+
    | Plugin | Core or optional? | Status |
    +-----------------------------------+-------------------+---------------+
    | Diagnostics | Core | Activated |
    | Resolution | Core | Activated |
    | ScheduledReports | Core | Activated |
    | UserCountryMap | Core | Activated |
    | Live | Core | Activated |
    | PrivacyManager | Core | Activated |
    | ImageGraph | Core | Activated |
    | Annotations | Core | Activated |
    | MobileMessaging | Core | Activated |
    | Login | Core | Activated |
    | SegmentEditor | Core | Activated |
    | Insights | Core | Activated |
    | Morpheus | Core | Activated |
    | Contents | Core | Activated |
    | BulkTracking | Core | Activated |
    | DevicePlugins | Core | Activated |
    | CoreUpdater | Core | Activated |
    | Heartbeat | Core | Activated |
    | Marketplace | Core | Activated |
    | ProfessionalServices | Core | Activated |
    | UserId | Core | Activated |
    | CustomDimensions | Core | Activated |
    | DBStats | Core | Activated |
    | MobileAppMeasurable | Core | Activated |
    | IntranetMeasurable | Core | Activated |
    | TwoFactorAuth | Core | Activated |
    | GeoIp2 | Core | Activated |
    | TagManager | Core | Activated |
    | Tour | Core | Activated |
    | CustomJsTracker | Core | Activated |
    | PagePerformance | Core | Activated |
    | CoreConsole | Core | Activated |
    | Overlay | Core | Activated |
    | Feedback | Core | Activated |
    | LanguagesManager | Core | Activated |
    | CoreAdminHome | Core | Activated |
    | UsersManager | Core | Activated |
    | SitesManager | Core | Activated |
    | Installation | Core | Activated |
    | Monolog | Core | Activated |
    | Intl | Core | Activated |
    | CorePluginsAdmin | Core | Activated |
    | CoreHome | Core | Activated |
    | RssWidget | Core | Activated |
    | CoreVisualizations | Core | Activated |
    | Proxy | Core | Activated |
    | API | Core | Activated |
    | Widgetize | Core | Activated |
    | Transitions | Core | Activated |
    | WebsiteMeasurable | Core | Activated |
    | Actions | Core | Activated |
    | Ecommerce | Core | Activated |
    | VisitorInterest | Core | Activated |
    | VisitTime | Core | Activated |
    | VisitFrequency | Core | Activated |
    | VisitsSummary | Core | Activated |
    | Dashboard | Core | Activated |
    | UserCountry | Core | Activated |
    | Events | Core | Activated |
    | SEO | Core | Activated |
    | CoreVue | Core | Activated |
    | Goals | Core | Activated |
    | MultiSites | Core | Activated |
    | Referrers | Core | Activated |
    | UserLanguage | Core | Activated |
    | DevicesDetection | Core | Activated |
    | TreemapVisualization | Optional | Activated |
    | UsersFlow | Optional | Activated |
    | Migration | Optional | Activated |
    | IPReports | Optional | Activated |
    | CustomVariables | Optional | Activated |
    | WooCommerceAnalytics | Optional | Not activated |
    | LoginSaml | Optional | Activated |
    | Funnels | Optional | Activated |
    | MultiChannelConversionAttribution | Optional | Activated |
    | RollUpReporting | Optional | Activated |
    | LiveTab | Optional | Activated |
    | FormAnalytics | Optional | Activated |
    | ActivityLog | Optional | Activated |
    | MediaAnalytics | Optional | Activated |
    | WhiteLabel | Optional | Activated |
    | HeatmapSessionRecording | Optional | Activated |
    | SecurityInfo | Optional | Activated |
    | CustomReports | Optional | Activated |
    | SearchEngineKeywordsPerformance | Optional | Activated |
    | Bandwidth | Optional | Not activated |
    | LogViewer | Optional | Activated |
    | Provider | Optional | Activated |
    | AbTesting | Optional | Activated |
    +-----------------------------------+-------------------+---------------+

  • Browser:
  • Operating System:
@grzchr15 commented on November 22nd 2022

May be linked to any changes happened which cause also https://github.com/matomo-org/matomo/issues/19695

@bx80 commented on November 23rd 2022 Contributor

Hi @grzchr15, thanks for reporting this.

Issue 19695 occurs when archiving data so it's probably not related.

Deleting log data can use quite a bit of memory. The code that ran out of memory in the fatal error above was building a database query with multiple visitor ids. It's possible that this area of code could be made more memory efficient.

Does decreasing the --limit parameter value allow this to complete for you?
Is it possible for you to increase the allowed memory for the process beyond 1.56GiB?

@grzchr15 commented on November 29th 2022

retried it with --100 does not change anything

@bx80 commented on November 29th 2022 Contributor

Thanks for trying that @grzchr15. I'll allocate this issue for prioritization.

Increasing the allowed PHP memory or breaking up the date ranges are probably the best options to get this to run for now.

Powered by GitHub Issue Mirror