@JRPonte opened this Issue on August 31st 2022

Hi all,
Since I updated my installation of matomo to version 4.11, the archive process of one of the sites that matomo is tracking has been systematically giving "Allowed memory exhausted" errors.
I've already increased PHP's memory to 8GB, start servers are 50, max children 100, and max requests 5000.
I've disabled archiving through the browser, and I've one cron job that runs every 5 minutes.
Matomo currently logs 35 sites and an average os 750k pageviews per day.

If the errors are limited to only 1 site, how can it be corrected? Should all archives be invalidated so that matomo can run the archive process again?

Matomo Version: 4.11.0
PHP Version: 7.3.28
MySQL: 10.3.29-MariaDB
Server OS: Debian 10.12
Server: Azure's standard D4as v4 (4 vcpus, 16 GiB memory)
CPU usage is normally at 90%+ during office hours, and drops to 30%+ outside that

Error log:
Error unserializing the following response from ?module=API&method=CoreAdminHome.archiveReports&idSite=34&period=day&date=2022-08-18&format=json&trigger=archivephp: 'PHP Fatal error: Allowed memory size of 8588886016 bytes exhausted (tried to allocate 7627087104 bytes) in /opt/bitnami/matomo/libs/Zend/Db/Statement/Mysqli.php on line 250 {"result":"error","message":"Allowed memory size of 8588886016 bytes exhausted (tried to allocate 7627087104 bytes) on \/opt\/bitnami\/matomo\/libs\/Zend\/Db\/Statement\/Mysqli.php(250) #0 \/opt\/bitnami\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(164): Piwik\Plugins\Actions\Archiver->callAggregateDayReport() ","backtrace":"Allowed memory size of 8588886016 bytes exhausted (tried to allocate 7627087104 bytes) on \/opt\/bitnami\/matomo\/libs\/Zend\/Db\/Statement\/Mysqli.php(250)\n#0 \/opt\/bitnami\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(164): Piwik\Plugins\Actions\Archiver->callAggregateDayReport()\n\n#0 [internal function]: Piwik\Plugins\CorePluginsAdmin\Controller->safemode(Array)\n#1 \/opt\/bitnami\/matomo\/core\/FrontController.php(631): call_user_func_array(Array, Array)\n#2 \/opt\/bitnami\/matomo\/core\/FrontController.php(169): Piwik\FrontController->doDispatch('CorePluginsAdmi...', 'safemode', Array)\n#3 \/opt\/bitnami\/matomo\/core\/FrontController.php(100): Piwik\FrontController->dispatch('CorePluginsAdmi...', 'safemode', Array)\n#4 \/opt\/bitnami\/matomo\/core\/FrontController.php(275): Piwik\FrontController::generateSafeModeOutputFromError(Array)\n#5 [internal function]: Piwik\FrontController::triggerSafeModeWhenError()\n#6 {main}"}{"result":"error","message":"Allowed memory size of 8588886016 bytes exhausted (tried to allocate 7627087104 bytes) on \/opt\/bitnami\/matomo\/libs\/Zend\/Db\/Statement\/Mysqli.php(250) #0 \/opt\/bitnami\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(164): Piwik\Plugins\Actions\Archiver->callAggregateDayReport() ","backtrace":"Allowed memory size of 8588886016 bytes exhausted (tried to allocate 7627087104 bytes) on \/opt\/bitnami\/matomo\/libs\/Zend\/Db\/Statement\/Mysqli.php(250)\n#0 \/opt\/bitnami\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(164): Piwik\Plugins\Actions\Archiver->callAggregateDayReport()\n\n#0 [internal function]: Piwik\Plugins\CorePluginsAdmin\Controller->safemode(Array)\n#1 \/opt\/bitnami\/matomo\/core\/FrontController.php(631): call_user_func_array(Array, Array)\n#2 \/opt\/bitnami\/matomo\/core\/FrontController.php(169): Piwik\FrontController->doDispatch('CorePluginsAdmi...', 'safemode', Array)\n#3 \/opt\/bitnami\/matomo\/core\/FrontController.php(100): Piwik\FrontController->dispatch('CorePluginsAdmi...', 'safemode', Array)\n#4 \/opt\/bitnami\/matomo\/core\/FrontController.php(275): Piwik\FrontController::generateSafeModeOutputFromError(Array)\n#5 [internal function]: Piwik\FrontController::triggerSafeModeWhenError()\n#6 {main}"}'

Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=34&period=day&date=2022-08-18&format=json&trigger=archivephp. Response was 'PHP Fatal error: Allowed memory size of 8588886016 bytes exhausted (tried to allocate 7627087104 bytes) in /opt/bitnami/matomo/libs/Zend/Db/Statement/Mysqli.php on line 250 {"result":"error","message":"Allowed memory size of 8588886016 bytes exhausted (tried to allocate 7627087104 bytes) on \/opt\/bitnami\/matomo\/libs\/Zend\/Db\/Statement\/Mysqli.php(250) #0 \/opt\/bitnami\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(164): Piwik\Plugins\Actions\Archiver->callAggregateDayReport() ","backtrace":"Allowed memory size of 8588886016 bytes exhausted (tried to allocate 7627087104 bytes) on \/opt\/bitnami\/matomo\/libs\/Zend\/Db\/Statement\/Mysqli.php(250)\n#0 \/opt\/bitnami\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(164): Piwik\Plugins\Actions\Archiver->callAggregateDayReport()\n\n#0 [internal function]: Piwik\Plugins\CorePluginsAdmin\Controller->safemode(Array)\n#1 \/opt\/bitnami\/matomo\/core\/FrontController.php(631): call_user_func_array(Array, Array)\n#2 \/opt\/bitnami\/matomo\/core\/FrontController.php(169): Piwik\FrontController->doDispatch('CorePluginsAdmi...', 'safemode', Array)\n#3 \/opt\/bitnami\/matomo\/core\/FrontController.php(100): Piwik\FrontController->dispatch('CorePluginsAdmi...', 'safemode', Array)\n#4 \/opt\/bitnami\/matomo\/core\/FrontController.php(275): Piwik\FrontController::generateSafeModeOutputFromError(Array)\n#5 [internal function]: Piwik\FrontController::triggerSafeModeWhenError()\n#6 {main}"}{"result":"error","message":"Allowed memory size of 8588886016 bytes exhausted (tried to allocate 7627087104 bytes) on \/opt\/bitnami\/matomo\/libs\/Zend\/Db\/Statement\/Mysqli.php(250) #0 \/opt\/bitnami\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(164): Piwik\Plugins\Actions\Archiver->callAggregateDayReport() ","backtrace":"Allowed memory size of 8588886016 bytes exhausted (tried to allocate 7627087104 bytes) on \/opt\/bitnami\/matomo\/libs\/Zend\/Db\/Statement\/Mysqli.php(250)\n#0 \/opt\/bitnami\/matomo\/core\/ArchiveProcessor\/PluginsArchiver.php(164): Piwik\Plugins\Actions\Archiver->callAggregateDayReport()\n\n#0 [internal function]: Piwik\Plugins\CorePluginsAdmin\Controller->safemode(Array)\n#1 \/opt\/bitnami\/matomo\/core\/FrontController.php(631): call_user_func_array(Array, Array)\n#2 \/opt\/bitnami\/matomo\/core\/FrontController.php(169): Piwik\FrontController->doDispatch('CorePluginsAdmi...', 'safemode', Array)\n#3 \/opt\/bitnami\/matomo\/core\/FrontController.php(100): Piwik\FrontController->dispatch('CorePluginsAdmi...', 'safemode', Array)\n#4 \/opt\/bitnami\/matomo\/core\/FrontController.php(275): Piwik\FrontController::generateSafeModeOutputFromError(Array)\n#5 [internal function]: Piwik\FrontController::triggerSafeModeWhenError()\n#6 {main}"}'

@mikkeschiren commented on August 31st 2022 Contributor

We also see much more need for RAM since upgrading our setups, with a lot of traffic.

@sgiehl commented on September 2nd 2022 Member

I assume this might most likely be caused by this new feature: https://github.com/matomo-org/matomo/pull/18221

@JRPonte commented on September 3rd 2022

Can I do something to prevent this errors? Or should I just wait for the next update?

@OlliWu commented on September 5th 2022

Just want to mention that we're also facing the same issue. We're tracking a few sites with lots of traffic. But only the one with a handfull of Events and Goals defined using a huge amount of Memory during archiving. The Process have peaks of 15-20 GB RAM depending on segment and period. We tweaked all necessary parameters to prevent errors. But monthly reports are still likely to fail.

@sgiehl commented on September 5th 2022 Member

As mentioned above, this is most likely caused by https://github.com/matomo-org/matomo/pull/18221.
We are aware that archiving consumes a lot more memory with that. We are currently evaluating possible improvements.
Maybe @bx80 has more insights on that...

@djmaze commented on September 16th 2022

Since upgrading, we are also not able to successfully run archive jobs on our server because of a hugely increased memory usage.

Is it possible to "disable" the new feature somehow? Any other workaround?

@bx80 commented on September 20th 2022 Contributor

Sorry about the delay with this. The next release (currently in release candidate status) will include a config option to disable archiving of action goal conversion data. We're looking at options to improve performance and reduce memory usage for this feature on larger datasets.

@bx80 commented on October 6th 2022 Contributor

Matomo 4.12 has been released and includes an option to disable archiving of action goal conversion data.
Adding disable_archive_actions_goals = 1 to the [General'] section of your config.ini.php will disable this feature entirely.

We're continuing to explore performance improvements for this feature so it can work well on large datasets with millions of conversions per day.

@djmaze commented on October 18th 2022

Unfortunately, even with disable_archive_actions_goals = 1 the archive process is still failing over here.

Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=5&period=year&date=2022-01-01&format=json&trigger=archivephp. Response was '{"result":"error","message":"Allowed memory size of 3246391296 bytes exhausted (tried to allocate 20480 bytes) on \/var\/www\/html\/core\/DataTable\/Row.php(361) <a href='/0'>#0</a> \/var\/www\/html\/core\/ArchiveProcessor.php(226): Piwik\\ArchiveProcessor-&gt;aggregateDataTableRecord(name=&quot;Actions_actions_url&quot;) <a href='/1'>#1</a> \/var\/www\/html\/core\/ArchiveProcessor\/PluginsArchiver.php(168): Piwik\\Plugins\\Actions\\Archiver-&gt;callAggregateMultipleReports()
[...]

UPDATE: With PHP_MEMORY_LIMIT increased to 4GB in the container, the archive run was finally successful.

@grzchr15 commented on November 21st 2022

Matomo version 4.12.3
Adding disable_archive_actions_goals = 1 to the [General'] section of your config.ini.php will disable this feature entirely.
Still
Allowed memory size of 1677721600 bytes exhausted (tried to allocate 20480 bytes) in /var/www/xx/matomo/core/DataTable.php on line 1483'

@djmaze commented on January 24th 2023

As a further feedback: shortly after my success report, memory was being exhausted again. It does not run successfully anymore.

Are there any developments to improve the memory usage? Otherwise we might be doomed using an only partially working analytics suite. And this is for a small- to medium-sized application.

Update: This time we had to increase memory size to 5 GB for the archive to succeed. If the memory usage will be ever growing, this is not a sustainable approach.

Powered by GitHub Issue Mirror