Big memory usage during archive #18460

Process of archiving sometimes is taking too much memory:

Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 25695760 bytes) in /usr/local/www/matomo/libs/Zend/Db/Statement/Pdo.php on line 233 Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 16777224 bytes) in /usr/local/www/matomo/core/DataTable/Manager.php on line 98 ' INFO [2021-12-06 04:00:25] 99708 Error: Error unserializing the following response from ?module=API&method=CoreAdminHome.archiveReports&idSite=23&period=year&date=2021-01-01&format=json&trigger=archivephp: ' Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 25695760 bytes) in /usr/local/www/matomo/libs/Zend/Db/Statement/Pdo.php on line 233 Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 16777224 bytes) in /usr/local/www/matomo/core/DataTable/Manager.php on line 98 ' INFO [2021-12-06 04:00:25] 99708 Error: Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=25&period=year&date=2021-01-01&format=json&trigger=archivephp. Response was ' Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 323584 bytes) in /usr/local/www/matomo/libs/Zend/Db/Statement/Pdo.php on line 233 Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 8388616 bytes) in /usr/local/www/matomo/core/DataTable/Manager.php on line 98 ' INFO [2021-12-06 04:00:25] 99708 Error: Error unserializing the following response from ?module=API&method=CoreAdminHome.archiveReports&idSite=25&period=year&date=2021-01-01&format=json&trigger=archivephp: ' Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 323584 bytes) in /usr/local/www/matomo/libs/Zend/Db/Statement/Pdo.php on line 233 Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 8388616 bytes) in /usr/local/www/matomo/core/DataTable/Manager.php on line 98

The limit of RAM for PHP is 8G but errors also appear (but less) with 12G.
Command for archiving I'm running:
php72 console core:archive --force-all-websites --php-cli-options='-dmemory_limit=8G'

Expected Behavior

Less RAM usage ;)

Your Environment

• Matomo Version: 4.5.0
• PHP Version: 7.2.34
• Server Operating System: FreeBSD 12/13
• Additionally installed plugins:
• Browser:
• Operating System:

Hi @avkarenow, thanks for reporting this issue, we're always looking to reduce memory usage. Are you archiving a large number of websites at once?

Matomo 4.7.0 will include https://github.com/matomo-org/matomo/pull/18326, a community submission which provides extra options for the archive command to split up the archiving task by a number of websites or reports. This can be used to reduce the memory usage that can occur with one long running process.

Also would be good to know if you have the same issue in the latest version, currently 4.6.1?

Sorry for my late reply.
Unfortunately, the problem with big RAM usage still appears.
Now I'm using Matomo 4.7.1

@bx80 I'm archiving 80-400 websites at once. I run the process once per day.

I'm experiencing a memory leak while archiving segments. I suspect a (premium) plugin to be responsible for that, because I have other, much bigger instances, that are not affected. However, I cannot tell, which one is responsible.
Creating the yearly archive for 2022 needs in peak more than 40G, while there are, for the whole Matomo instance, just 1.5M Actions for 2022. The memory usage seems to be very high for every segment, some might need more than others (but I didn't find a pattern yet).

Matomo is Version 4.6.2
PHP-CLI: 8.0
Matomo instance traffic: 1.5M Actions this year

Active plugins: API, AbTesting 4.1.6, Actions, ActivityLog 4.0.5, Annotations, BulkTracking, Cohorts 4.0.5, Contents, CoreAdminHome, CoreConsole, CoreHome, CorePluginsAdmin, CoreUpdater, CoreVisualizations, CoreVue, CustomAlerts 4.0.4, CustomDimensions, CustomJsTracker, CustomReports 4.0.12, CustomVariables 4.0.1, DBStats, Dashboard, DeviceDetectorCache 4.2.3, DevicePlugins, DevicesDetection, Diagnostics, Ecommerce, Events, FormAnalytics 4.0.8, Funnels 4.0.10, GKVTrackingLocalPrevention 1.0, GeoIp2, Goals, Heartbeat, HeatmapSessionRecording 4.4.2, ImageGraph, Insights, Installation, Intl, IntranetMeasurable, InvalidateReports 4.0.1, LanguagesManager, Live, LogViewer 4.0.1, Login, MarketingCampaignsReporting 4.1.1, Marketplace, MediaAnalytics 4.0.17, MobileAppMeasurable, MobileMessaging, Monolog, Morpheus, MultiChannelConversionAttribution 4.0.5, MultiSites, Overlay, PagePerformance, PrivacyManager, Provider 4.0.3, Proxy, QueuedTracking 4.0.2, Referrers, Resolution, RollUpReporting 4.0.3, ScheduledReports, SearchEngineKeywordsPerformance 4.3.8, SegmentEditor, SitesManager, TagManager, Transitions, UserCountry, UserCountryMap, UserId, UserLanguage, UsersFlow 4.0.4, UsersManager, VisitFrequency, VisitTime, VisitorInterest, VisitsSummary, WebsiteMeasurable, Widgetize, WooCommerceAnalytics 4.0.5

@avkarenow Do you also have any premium plugins installed?

From additional plugins I'm using only https://plugins.matomo.org/LoginTokenAuth

Did you solve this issue? I'm having the same issue right now, with Matomo 3.14.1 16GB were enough, now I'm running 4.10.1 and not even having 20% of the archiving completed, it already consumes more than 32gb and aborts due to the memory limit. I disabled the CustomDimensions plugin, the memory usage is still insane tho, using 32gb not even 33% through the archiving. Is there anything one can do? I dont think its normal to require such insane amounts of memory.

Hi @Pflegusch, thanks for reaching out. Unfortunately this issue is still ongoing as it seems to be dependent on a number of factors and is proving difficult to recreate reliably. As a workaround you could try the max-archives-to-process option detailed here: https://matomo.org/faq/how-do-i-decrease-archiving-memory-usage-when-processing-a-large-number-of-websites/

Hi @bx80, thank you for the reply. I tried the --force-date-range parameter now, however, I'm not really sure if it really does what I'm thinking. I created a script which calls core:archive for the last 5 years, so I did something like this:

/usr/local/bin/php /var/www/html/console core:archive --force-date-range 2018-01-01,2018-12-31 --matomo-domain=https://$MATOMO_URL > /var/tmp/piwik-ego-archive-2018.log
/usr/local/bin/php /var/www/html/console core:archive --force-date-range 2019-01-01,2019-12-31 --matomo-domain=https://$MATOMO_URL > /var/tmp/piwik-ego-archive-2019.log
/usr/local/bin/php /var/www/html/console core:archive --force-date-range 2020-01-01,2020-12-31 --matomo-domain=https://$MATOMO_URL > /var/tmp/piwik-ego-archive-2020.log
/usr/local/bin/php /var/www/html/console core:archive --force-date-range 2021-01-01,2021-12-31 --matomo-domain=https://$MATOMO_URL > /var/tmp/piwik-ego-archive-2021.log
/usr/local/bin/php /var/www/html/console core:archive --force-date-range 2022-01-01,2022-12-31 --matomo-domain=https://$MATOMO_URL > /var/tmp/piwik-ego-archive-2022.log

The memory consumption now is much, much smaller (around 350 MB), but is it really the correct way of doing so? The logs above look suspiciously the same and the process takes around an hour for all of them. I'm not sure if I did it correctly. Otherwise I would try to use the --max-archives-to-process argument.