Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

archive process trouble with parameter N #16409

Closed
Jinzol opened this issue Sep 8, 2020 · 7 comments
Closed

archive process trouble with parameter N #16409

Jinzol opened this issue Sep 8, 2020 · 7 comments
Labels
answered For when a question was asked and we referred to forum or answered it.

Comments

@Jinzol
Copy link

Jinzol commented Sep 8, 2020

Hi I want to process some specific data for a specific segment for specific time.
It seems the parameter " --force-date-last-n" does not work anymore and default is used.

The code what I use is
/opt/rh/rh-php72/root/usr/bin/php /srv/www/matomo/console core:archive --force-date-last-n=45 --force-idsites=3 --force-periods=day --force-idsegments=450 --concurrent-requests-per-website=1

When I check the logging of this piece of code it shows
INFO [2020-09-08 15:34:37] 26330 Will pre-process for website id = 3, period = day, date = last2 INFO [2020-09-08 15:34:37] 26330 - pre-processing all visits
It should be last45 in this case and it has worked.

We run Matomo 3.13.5

How can I force to run the previous 45 days instead the last 2. Date range is not really an option.

I hope you can help me :-)

@tsteur
Copy link
Member

tsteur commented Sep 8, 2020

Hi @Jinzol thanks for creating the issue.

Did you invalidate the reports before running this command? AFAIK we added some code to change lastX on demand eg from last45 to last2 should all archives be finished and none are invalidated.

@Jinzol
Copy link
Author

Jinzol commented Sep 9, 2020

Well I did not invalidate the reports before running because I want to use a custom N with archiving. I have problem that the data of the month august is not correct while the archives says its correct. We duplicate the data to an other environment with its own archiving process and there is a difference in results. What I think what is happening is that my archiving process takes sometimes to long (48h+) and the next archiver run will skip data because N is to small. You get small gaps in the processed data and that gives different results. Thats why I want to always increase the value of N. But that my theorie :-)

@tsteur
Copy link
Member

tsteur commented Sep 9, 2020

Well I did not invalidate the reports before running because I want to use a custom N with archiving

If you don't invalidate the reports upfront, Matomo should usually not rearchive these reports even if it uses eg last45. Even if it was using last45, Matomo should then see there are existing archives for most days and ignore it. It would only rearchive them if the archives for these days never existed. I'm not sure if this procedure was done before but thinking it shouldn't really have worked before unless for some reasons the archives were never finished for those days where the data wasn't correct (eg month of August). I'd definitely recommend to first invalidate the data @Jinzol . Could you give that a try?

@Jinzol
Copy link
Author

Jinzol commented Sep 10, 2020

Well I can give that a try and I will report it back. I takes some time to reprocess the data. But in the meantime I still don't understand why the archivers have finished correctly and the processed data is wrong. Or my raw data is incorrect but with the mirror environment it seems that is not the issue. We have seen more deadlocks on the database. What kind of effect does that have of the archivers? Or how can I prevent them. I would like to say to my colleague's the archiver is done and the data is trustworthy. The problem with invalidate reports is that I know something is wrong and now I don't know what went wrong.

@tsteur
Copy link
Member

tsteur commented Sep 13, 2020

@Jinzol let us know when you have an update. It's hard to say if/what went wrong there. It also depends maybe how it all works with your mirror environment etc. Is it happening repeatedly or just once?

What kind of deadlocks are triggered? Do you have more information eg what kind of query is having the issue?

@Jinzol
Copy link
Author

Jinzol commented Sep 15, 2020

Hi @tsteur, well it took some time but I reprocessed august and it seems fixed.
I started with

  • day with n=45
  • day,week n=8
  • day,week,month n=2

The last one took 80768 seconds or 22,43 hours. I hope it gets abit faster with day,week,month and n=2. I think my problem started that some archive run didn't finish correctly. I will add the logging of the deadlock.

archive_website_except_3_20200914_1445:INFO [2020-09-14 13:05:52] 3213 Error: Got invalid response from API request: ?module=API&method=API.get&idSite=88&period=day&date=last2&format=php&trigger=archivephp. Response was 'a:2:{s:6:"result";s:5:"error";s:7:"message";s:20624:"SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction - in plugin Funnels #0 /srv/www/matomo/core/ArchiveProcessor/Loader.php(165): Piwik\ArchiveProcessor\PluginsArchiver->callAggregateAllPlugins('1324', '787', false) #1 /srv/www/matomo/core/ArchiveProcessor/Loader.php(111): Piwik\ArchiveProcessor\Loader->prepareAllPluginsArchive('1324', '787') #2 /srv/www/matomo/core/ArchiveProcessor/Loader.php(84): Piwik\ArchiveProcessor\Loader->prepareArchiveImpl('VisitsSummary') #3 /srv/www/matomo/core/Context.php(75): Piwik\ArchiveProcessor\Loader->Piwik\ArchiveProcessor\{closure}() #4 /srv/www/matomo/core/ArchiveProcessor/Loader.php(85): Piwik\Context::changeIdSite(48, Object(Closure)) #5 /srv/www/matomo/core/Archive.php(803): Piwik\ArchiveProcessor\Loader->prepareArchive('VisitsSummary') #6 /srv/www/matomo/core/Archive.php(607): Piwik\Archive->prepareArchive(Array, Object(Piwik\Site), Object(Piwik\Period\Day)) #7 /srv/www/matomo/core/Archive.php(551): Piwik\Archive->cacheArchiveIdsAfterLaunching(Array, Array) #8 /srv/www/matomo/core/Archive.php(480): Piwik\Archive->getArchiveIds(Array) #9 /srv/www/matomo/core/Archive.php(295): Piwik\Archive->get(Array, 'numeric') #10 /srv/www/matomo/core/ArchiveProcessor.php(602): Piwik\Archive->getDataTableFromNumeric(Array) #11 /srv/www/matomo/core/ArchiveProcessor.php(247): Piwik\ArchiveProcessor->getAggregatedNumericMetrics(Array, false) #12 /srv/www/matomo/core/ArchiveProcessor/PluginsArchiver.php(307): Piwik\ArchiveProcessor->aggregateNumericMetrics(Array) #13 /srv/www/matomo/core/ArchiveProcessor/PluginsArchiver.php(108): Piwik\ArchiveProcessor\PluginsArchiver->aggregateMultipleVisitsMetrics() #14 /srv/www/matomo/core/ArchiveProcessor/Loader.php(159): Piwik\ArchiveProcessor\PluginsArchiver->callAggregateCoreMetrics() #15 /srv/www/matomo/core/ArchiveProcessor/Loader.php(111): Piwik\ArchiveProcessor\Loader->prepareAllPluginsArchive('15255', '13984') #16 /srv/www/matomo/core/ArchiveProcessor/Loader.php(84): Piwik\ArchiveProcessor\Loader->prepareArchiveImpl('VisitsSummary') #17 /srv/www/matomo/core/Context.php(75): Piwik\ArchiveProcessor\Loader->Piwik\ArchiveProcessor\{closure}() #18 /srv/www/matomo/core/ArchiveProcessor/Loader.php(85): Piwik\Context::changeIdSite(88, Object(Closure)) #19 /srv/www/matomo/core/Archive.php(803): Piwik\ArchiveProcessor\Loader->prepareArchive('VisitsSummary') #20 /srv/www/matomo/core/Archive.php(607): Piwik\Archive->prepareArchive(Array, Object(Piwik\Site), Object(Piwik\Period\Day)) #21 /srv/www/matomo/core/Archive.php(551): Piwik\Archive->cacheArchiveIdsAfterLaunching(Array, Array) #22 /srv/www/matomo/core/Archive.php(480): Piwik\Archive->getArchiveIds(Array) #23 /srv/www/matomo/core/Archive.php(295): Piwik\Archive->get(Array, 'numeric') #24 /srv/www/matomo/plugins/VisitsSummary/API.php(36): Piwik\Archive->getDataTableFromNumeric(Array) #25 [internal function]: Piwik\Plugins\VisitsSummary\API->get('88', 'day', 'last2', false, Array) #26 /srv/www/matomo/core/API/Proxy.php(237): call_user_func_array(Array, Array) #27 /srv/www/matomo/core/Context.php(28): Piwik\API\Proxy->Piwik\API\{closure}() #28 /srv/www/matomo/core/API/Proxy.php(328): Piwik\Context::executeWithQueryParameters(Array, Object(Closure)) #29 /srv/www/matomo/plugins/API/API.php(437): Piwik\API\Proxy->call('\\Piwik\\Plugins\\...', 'get', Array) #30 [internal function]: Piwik\Plugins\API\API->get('88', 'day', 'last2', false, Array) #31 /srv/www/matomo/core/API/Proxy.php(237): call_user_func_array(Array, Array) #32 /srv/www/matomo/core/Context.php(28): Piwik\API\Proxy->Piwik\API\{closure}() #33 /srv/www/matomo/core/API/Proxy.php(328): Piwik\Context::executeWithQueryParameters(Array, Object(Closure)) #34 /srv/www/matomo/core/API/Request.php(266): Piwik\API\Proxy->call('\\Piwik\\Plugins\\...', 'get', Array) #35 /srv/www/matomo/plugins/API/Controller.php(41): Piwik\API\Request->process() #36 [internal function]: Piwik\Plugins\API\Controller->index() #37 /srv/www/matomo/core/FrontController.php(590): call_user_func_array(Array, Array) #38 /srv/www/matomo/core/FrontController.php(165): Piwik\FrontController->doDispatch('API', false, Array) #39 /srv/www/matomo/core/dispatch.php(34): Piwik\FrontController->dispatch() #40 /srv/www/matomo/index.php(27): require_once('/srv/www/matomo...') #41 /srv/www/matomo/core/CliMulti/RequestCommand.php(79): require_once('/srv/www/matomo...') #42 /srv/www/matomo/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\CliMulti\RequestCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #43 /srv/www/matomo/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #44 /srv/www/matomo/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application->doRunCommand(Object(Piwik\CliMulti\RequestCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #45 [internal function]: Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #46 /srv/www/matomo/core/Console.php(140): call_user_func(Array, Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput)) #47 /srv/www/matomo/core/Access.php(644): Piw ... ut)) #59 /srv/www/matomo/console(32): Symfony\Component\Console\Application->run() #60 {main}";}'

It seems it happens around ~5 a day.

@tsteur
Copy link
Member

tsteur commented Sep 16, 2020

Thanks. Be good to create a new issue for the deadlock if there isn't one already for that particular deadlock (we can also close as duplicate if needed). be great to watch if the problem happens again, otherwise we can maybe close the issue now and reopen if the problem happens again?

You will maybe also want to write the output of the archiving somewhere like

1 * * * * php path/to/matomo/console core:archive > archive.log

@Jinzol Jinzol closed this as completed Sep 16, 2020
@tsteur tsteur added the answered For when a question was asked and we referred to forum or answered it. label Sep 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
answered For when a question was asked and we referred to forum or answered it.
Projects
None yet
Development

No branches or pull requests

2 participants