@Jinzol opened this Issue on September 8th 2020

Hi I want to process some specific data for a specific segment for specific time.
It seems the parameter " --force-date-last-n" does not work anymore and default is used.

The code what I use is
/opt/rh/rh-php72/root/usr/bin/php /srv/www/matomo/console core:archive --force-date-last-n=45 --force-idsites=3 --force-periods=day --force-idsegments=450 --concurrent-requests-per-website=1

When I check the logging of this piece of code it shows
INFO [2020-09-08 15:34:37] 26330 Will pre-process for website id = 3, period = day, date = last2 INFO [2020-09-08 15:34:37] 26330 - pre-processing all visits
It should be last45 in this case and it has worked.

We run Matomo 3.13.5

How can I force to run the previous 45 days instead the last 2. Date range is not really an option.

I hope you can help me :-)

@tsteur commented on September 8th 2020 Member

Hi @Jinzol thanks for creating the issue.

Did you invalidate the reports before running this command? AFAIK we added some code to change lastX on demand eg from last45 to last2 should all archives be finished and none are invalidated.

@Jinzol commented on September 9th 2020

Well I did not invalidate the reports before running because I want to use a custom N with archiving. I have problem that the data of the month august is not correct while the archives says its correct. We duplicate the data to an other environment with its own archiving process and there is a difference in results. What I think what is happening is that my archiving process takes sometimes to long (48h+) and the next archiver run will skip data because N is to small. You get small gaps in the processed data and that gives different results. Thats why I want to always increase the value of N. But that my theorie :-)

@tsteur commented on September 9th 2020 Member

Well I did not invalidate the reports before running because I want to use a custom N with archiving

If you don't invalidate the reports upfront, Matomo should usually not rearchive these reports even if it uses eg last45. Even if it was using last45, Matomo should then see there are existing archives for most days and ignore it. It would only rearchive them if the archives for these days never existed. I'm not sure if this procedure was done before but thinking it shouldn't really have worked before unless for some reasons the archives were never finished for those days where the data wasn't correct (eg month of August). I'd definitely recommend to first invalidate the data @Jinzol . Could you give that a try?

@Jinzol commented on September 10th 2020

Well I can give that a try and I will report it back. I takes some time to reprocess the data. But in the meantime I still don't understand why the archivers have finished correctly and the processed data is wrong. Or my raw data is incorrect but with the mirror environment it seems that is not the issue. We have seen more deadlocks on the database. What kind of effect does that have of the archivers? Or how can I prevent them. I would like to say to my colleague's the archiver is done and the data is trustworthy. The problem with invalidate reports is that I know something is wrong and now I don't know what went wrong.

@tsteur commented on September 13th 2020 Member

@Jinzol let us know when you have an update. It's hard to say if/what went wrong there. It also depends maybe how it all works with your mirror environment etc. Is it happening repeatedly or just once?

What kind of deadlocks are triggered? Do you have more information eg what kind of query is having the issue?

@Jinzol commented on September 15th 2020

Hi @tsteur, well it took some time but I reprocessed august and it seems fixed.
I started with

  • day with n=45
  • day,week n=8
  • day,week,month n=2

The last one took 80768 seconds or 22,43 hours. I hope it gets abit faster with day,week,month and n=2. I think my problem started that some archive run didn't finish correctly. I will add the logging of the deadlock.

archive_website_except_3_20200914_1445:INFO [2020-09-14 13:05:52] 3213 Error: Got invalid response from API request: ?module=API&method=API.get&idSite=88&period=day&date=last2&format=php&trigger=archivephp. Response was 'a:2:{s:6:"result";s:5:"error";s:7:"message";s:20624:"SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction - in plugin Funnels <a href='/0'>#0</a> /srv/www/matomo/core/ArchiveProcessor/Loader.php(165): Piwik\ArchiveProcessor\PluginsArchiver-&gt;callAggregateAllPlugins('1324', '787', false) <a href='/1'>#1</a> /srv/www/matomo/core/ArchiveProcessor/Loader.php(111): Piwik\ArchiveProcessor\Loader-&gt;prepareAllPluginsArchive('1324', '787') <a href='/2'>#2</a> /srv/www/matomo/core/ArchiveProcessor/Loader.php(84): Piwik\ArchiveProcessor\Loader-&gt;prepareArchiveImpl('VisitsSummary') <a href='/3'>#3</a> /srv/www/matomo/core/Context.php(75): Piwik\ArchiveProcessor\Loader-&gt;Piwik\ArchiveProcessor\{closure}() <a href='/4'>#4</a> /srv/www/matomo/core/ArchiveProcessor/Loader.php(85): Piwik\Context::changeIdSite(48, Object(Closure)) <a href='/5'>#5</a> /srv/www/matomo/core/Archive.php(803): Piwik\ArchiveProcessor\Loader-&gt;prepareArchive('VisitsSummary') <a href='/6'>#6</a> /srv/www/matomo/core/Archive.php(607): Piwik\Archive-&gt;prepareArchive(Array, Object(Piwik\Site), Object(Piwik\Period\Day)) <a href='/7'>#7</a> /srv/www/matomo/core/Archive.php(551): Piwik\Archive-&gt;cacheArchiveIdsAfterLaunching(Array, Array) <a href='/8'>#8</a> /srv/www/matomo/core/Archive.php(480): Piwik\Archive-&gt;getArchiveIds(Array) <a href='/9'>#9</a> /srv/www/matomo/core/Archive.php(295): Piwik\Archive-&gt;get(Array, 'numeric') <a href='/10'>#10</a> /srv/www/matomo/core/ArchiveProcessor.php(602): Piwik\Archive-&gt;getDataTableFromNumeric(Array) <a href='/11'>#11</a> /srv/www/matomo/core/ArchiveProcessor.php(247): Piwik\ArchiveProcessor-&gt;getAggregatedNumericMetrics(Array, false) <a href='/12'>#12</a> /srv/www/matomo/core/ArchiveProcessor/PluginsArchiver.php(307): Piwik\ArchiveProcessor-&gt;aggregateNumericMetrics(Array) <a href='/13'>#13</a> /srv/www/matomo/core/ArchiveProcessor/PluginsArchiver.php(108): Piwik\ArchiveProcessor\PluginsArchiver-&gt;aggregateMultipleVisitsMetrics() <a href='/14'>#14</a> /srv/www/matomo/core/ArchiveProcessor/Loader.php(159): Piwik\ArchiveProcessor\PluginsArchiver-&gt;callAggregateCoreMetrics() <a href='/15'>#15</a> /srv/www/matomo/core/ArchiveProcessor/Loader.php(111): Piwik\ArchiveProcessor\Loader-&gt;prepareAllPluginsArchive('15255', '13984') <a href='/16'>#16</a> /srv/www/matomo/core/ArchiveProcessor/Loader.php(84): Piwik\ArchiveProcessor\Loader-&gt;prepareArchiveImpl('VisitsSummary') <a href='/17'>#17</a> /srv/www/matomo/core/Context.php(75): Piwik\ArchiveProcessor\Loader-&gt;Piwik\ArchiveProcessor\{closure}() <a href='/18'>#18</a> /srv/www/matomo/core/ArchiveProcessor/Loader.php(85): Piwik\Context::changeIdSite(88, Object(Closure)) <a href='/19'>#19</a> /srv/www/matomo/core/Archive.php(803): Piwik\ArchiveProcessor\Loader-&gt;prepareArchive('VisitsSummary') <a href='/20'>#20</a> /srv/www/matomo/core/Archive.php(607): Piwik\Archive-&gt;prepareArchive(Array, Object(Piwik\Site), Object(Piwik\Period\Day)) <a href='/21'>#21</a> /srv/www/matomo/core/Archive.php(551): Piwik\Archive-&gt;cacheArchiveIdsAfterLaunching(Array, Array) <a href='/22'>#22</a> /srv/www/matomo/core/Archive.php(480): Piwik\Archive-&gt;getArchiveIds(Array) <a href='/23'>#23</a> /srv/www/matomo/core/Archive.php(295): Piwik\Archive-&gt;get(Array, 'numeric') <a href='/24'>#24</a> /srv/www/matomo/plugins/VisitsSummary/API.php(36): Piwik\Archive-&gt;getDataTableFromNumeric(Array) <a href='/25'>#25</a> [internal function]: Piwik\Plugins\VisitsSummary\API-&gt;get('88', 'day', 'last2', false, Array) <a href='/26'>#26</a> /srv/www/matomo/core/API/Proxy.php(237): call_user_func_array(Array, Array) <a href='/27'>#27</a> /srv/www/matomo/core/Context.php(28): Piwik\API\Proxy-&gt;Piwik\API\{closure}() <a href='/28'>#28</a> /srv/www/matomo/core/API/Proxy.php(328): Piwik\Context::executeWithQueryParameters(Array, Object(Closure)) <a href='/29'>#29</a> /srv/www/matomo/plugins/API/API.php(437): Piwik\API\Proxy-&gt;call('\\Piwik\\Plugins\\...', 'get', Array) <a href='/30'>#30</a> [internal function]: Piwik\Plugins\API\API-&gt;get('88', 'day', 'last2', false, Array) <a href='/31'>#31</a> /srv/www/matomo/core/API/Proxy.php(237): call_user_func_array(Array, Array) <a href='/32'>#32</a> /srv/www/matomo/core/Context.php(28): Piwik\API\Proxy-&gt;Piwik\API\{closure}() <a href='/33'>#33</a> /srv/www/matomo/core/API/Proxy.php(328): Piwik\Context::executeWithQueryParameters(Array, Object(Closure)) <a href='/34'>#34</a> /srv/www/matomo/core/API/Request.php(266): Piwik\API\Proxy-&gt;call('\\Piwik\\Plugins\\...', 'get', Array) <a href='/35'>#35</a> /srv/www/matomo/plugins/API/Controller.php(41): Piwik\API\Request-&gt;process() <a href='/36'>#36</a> [internal function]: Piwik\Plugins\API\Controller-&gt;index() <a href='/37'>#37</a> /srv/www/matomo/core/FrontController.php(590): call_user_func_array(Array, Array) <a href='/38'>#38</a> /srv/www/matomo/core/FrontController.php(165): Piwik\FrontController-&gt;doDispatch('API', false, Array) <a href='/39'>#39</a> /srv/www/matomo/core/dispatch.php(34): Piwik\FrontController-&gt;dispatch() <a href='/40'>#40</a> /srv/www/matomo/index.php(27): require_once('/srv/www/matomo...') <a href='/41'>#41</a> /srv/www/matomo/core/CliMulti/RequestCommand.php(79): require_once('/srv/www/matomo...') <a href='/42'>#42</a> /srv/www/matomo/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\CliMulti\RequestCommand-&gt;execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) <a href='/43'>#43</a> /srv/www/matomo/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command-&gt;run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) <a href='/44'>#44</a> /srv/www/matomo/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application-&gt;doRunCommand(Object(Piwik\CliMulti\RequestCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) <a href='/45'>#45</a> [internal function]: Symfony\Component\Console\Application-&gt;doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) <a href='/46'>#46</a> /srv/www/matomo/core/Console.php(140): call_user_func(Array, Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) <a href='/47'>#47</a> /srv/www/matomo/core/Access.php(644): Piw ... ut)) <a href='/59'>#59</a> /srv/www/matomo/console(32): Symfony\Component\Console\Application-&gt;run() <a href='/60'>#60</a> {main}";}'

It seems it happens around ~5 a day.

@tsteur commented on September 16th 2020 Member

Thanks. Be good to create a new issue for the deadlock if there isn't one already for that particular deadlock (we can also close as duplicate if needed). be great to watch if the problem happens again, otherwise we can maybe close the issue now and reopen if the problem happens again?

You will maybe also want to write the output of the archiving somewhere like

1 * * * * php path/to/matomo/console core:archive > archive.log
This Issue was closed on September 16th 2020
Powered by GitHub Issue Mirror