@avkarenow opened this Issue on December 7th 2021 Contributor

Process of archiving sometimes is taking too much memory:

Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 25695760 bytes) in /usr/local/www/matomo/libs/Zend/Db/Statement/Pdo.php on line 233 Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 16777224 bytes) in /usr/local/www/matomo/core/DataTable/Manager.php on line 98 ' INFO [2021-12-06 04:00:25] 99708 Error: Error unserializing the following response from ?module=API&method=CoreAdminHome.archiveReports&idSite=23&period=year&date=2021-01-01&format=json&trigger=archivephp: ' Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 25695760 bytes) in /usr/local/www/matomo/libs/Zend/Db/Statement/Pdo.php on line 233 Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 16777224 bytes) in /usr/local/www/matomo/core/DataTable/Manager.php on line 98 ' INFO [2021-12-06 04:00:25] 99708 Error: Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=25&period=year&date=2021-01-01&format=json&trigger=archivephp. Response was ' Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 323584 bytes) in /usr/local/www/matomo/libs/Zend/Db/Statement/Pdo.php on line 233 Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 8388616 bytes) in /usr/local/www/matomo/core/DataTable/Manager.php on line 98 ' INFO [2021-12-06 04:00:25] 99708 Error: Error unserializing the following response from ?module=API&method=CoreAdminHome.archiveReports&idSite=25&period=year&date=2021-01-01&format=json&trigger=archivephp: ' Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 323584 bytes) in /usr/local/www/matomo/libs/Zend/Db/Statement/Pdo.php on line 233 Fatal error: Allowed memory size of 8589934592 bytes exhausted (tried to allocate 8388616 bytes) in /usr/local/www/matomo/core/DataTable/Manager.php on line 98

The limit of RAM for PHP is 8G but errors also appear (but less) with 12G.
Command for archiving I'm running:
php72 console core:archive --force-all-websites --php-cli-options='-dmemory_limit=8G'

Expected Behavior

Less RAM usage ;)

Your Environment

  • Matomo Version: 4.5.0
  • PHP Version: 7.2.34
  • Server Operating System: FreeBSD 12/13
  • Additionally installed plugins:
  • Browser:
  • Operating System:
@bx80 commented on December 7th 2021 Contributor

Hi @avkarenow, thanks for reporting this issue, we're always looking to reduce memory usage. Are you archiving a large number of websites at once?

Matomo 4.7.0 will include https://github.com/matomo-org/matomo/pull/18326, a community submission which provides extra options for the archive command to split up the archiving task by a number of websites or reports. This can be used to reduce the memory usage that can occur with one long running process.

@justinvelluppillai commented on December 7th 2021 Contributor

Also would be good to know if you have the same issue in the latest version, currently 4.6.1?

@avkarenow commented on March 12th 2022 Contributor

Sorry for my late reply.
Unfortunately, the problem with big RAM usage still appears.
Now I'm using Matomo 4.7.1

@bx80 I'm archiving 80-400 websites at once. I run the process once per day.

@peterbo commented on April 8th 2022 Contributor

I'm experiencing a memory leak while archiving segments. I suspect a (premium) plugin to be responsible for that, because I have other, much bigger instances, that are not affected. However, I cannot tell, which one is responsible.
Creating the yearly archive for 2022 needs in peak more than 40G, while there are, for the whole Matomo instance, just 1.5M Actions for 2022. The memory usage seems to be very high for every segment, some might need more than others (but I didn't find a pattern yet).

Matomo is Version 4.6.2
PHP-CLI: 8.0
Matomo instance traffic: 1.5M Actions this year

Active plugins: API, AbTesting 4.1.6, Actions, ActivityLog 4.0.5, Annotations, BulkTracking, Cohorts 4.0.5, Contents, CoreAdminHome, CoreConsole, CoreHome, CorePluginsAdmin, CoreUpdater, CoreVisualizations, CoreVue, CustomAlerts 4.0.4, CustomDimensions, CustomJsTracker, CustomReports 4.0.12, CustomVariables 4.0.1, DBStats, Dashboard, DeviceDetectorCache 4.2.3, DevicePlugins, DevicesDetection, Diagnostics, Ecommerce, Events, FormAnalytics 4.0.8, Funnels 4.0.10, GKVTrackingLocalPrevention 1.0, GeoIp2, Goals, Heartbeat, HeatmapSessionRecording 4.4.2, ImageGraph, Insights, Installation, Intl, IntranetMeasurable, InvalidateReports 4.0.1, LanguagesManager, Live, LogViewer 4.0.1, Login, MarketingCampaignsReporting 4.1.1, Marketplace, MediaAnalytics 4.0.17, MobileAppMeasurable, MobileMessaging, Monolog, Morpheus, MultiChannelConversionAttribution 4.0.5, MultiSites, Overlay, PagePerformance, PrivacyManager, Provider 4.0.3, Proxy, QueuedTracking 4.0.2, Referrers, Resolution, RollUpReporting 4.0.3, ScheduledReports, SearchEngineKeywordsPerformance 4.3.8, SegmentEditor, SitesManager, TagManager, Transitions, UserCountry, UserCountryMap, UserId, UserLanguage, UsersFlow 4.0.4, UsersManager, VisitFrequency, VisitTime, VisitorInterest, VisitsSummary, WebsiteMeasurable, Widgetize, WooCommerceAnalytics 4.0.5

memory_leak PNG

memory_leak1

memory_usage_archiver PNG

@sgiehl commented on April 11th 2022 Member

@avkarenow Do you also have any premium plugins installed?

@avkarenow commented on April 21st 2022 Contributor

From additional plugins I'm using only https://plugins.matomo.org/LoginTokenAuth

@Pflegusch commented on June 29th 2022

Did you solve this issue? I'm having the same issue right now, with Matomo 3.14.1 16GB were enough, now I'm running 4.10.1 and not even having 20% of the archiving completed, it already consumes more than 32gb and aborts due to the memory limit. I disabled the CustomDimensions plugin, the memory usage is still insane tho, using 32gb not even 33% through the archiving. Is there anything one can do? I dont think its normal to require such insane amounts of memory.

@bx80 commented on June 29th 2022 Contributor

Hi @Pflegusch, thanks for reaching out. Unfortunately this issue is still ongoing as it seems to be dependent on a number of factors and is proving difficult to recreate reliably. As a workaround you could try the max-archives-to-process option detailed here: https://matomo.org/faq/how-do-i-decrease-archiving-memory-usage-when-processing-a-large-number-of-websites/

@Pflegusch commented on July 4th 2022

Hi @bx80, thank you for the reply. I tried the --force-date-range parameter now, however, I'm not really sure if it really does what I'm thinking. I created a script which calls core:archive for the last 5 years, so I did something like this:

/usr/local/bin/php /var/www/html/console core:archive --force-date-range 2018-01-01,2018-12-31 --matomo-domain=https://$MATOMO_URL > /var/tmp/piwik-ego-archive-2018.log
/usr/local/bin/php /var/www/html/console core:archive --force-date-range 2019-01-01,2019-12-31 --matomo-domain=https://$MATOMO_URL > /var/tmp/piwik-ego-archive-2019.log
/usr/local/bin/php /var/www/html/console core:archive --force-date-range 2020-01-01,2020-12-31 --matomo-domain=https://$MATOMO_URL > /var/tmp/piwik-ego-archive-2020.log
/usr/local/bin/php /var/www/html/console core:archive --force-date-range 2021-01-01,2021-12-31 --matomo-domain=https://$MATOMO_URL > /var/tmp/piwik-ego-archive-2021.log
/usr/local/bin/php /var/www/html/console core:archive --force-date-range 2022-01-01,2022-12-31 --matomo-domain=https://$MATOMO_URL > /var/tmp/piwik-ego-archive-2022.log

The memory consumption now is much, much smaller (around 350 MB), but is it really the correct way of doing so? The logs above look suspiciously the same and the process takes around an hour for all of them. I'm not sure if I did it correctly. Otherwise I would try to use the --max-archives-to-process argument.

Powered by GitHub Issue Mirror