New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Do not run the same archive command twice #12702
Conversation
FYI: https://github.com/matomo-org/matomo/blob/archivenottwice/core/CronArchive.php#L705-L716 might cause some trouble in that possibly a weekly archive will be skipped - because the weekly is already in process but not finished - so it already starts the monthly. However, the weekly is not finished. This has been a problem for a while when eg a weekly archive fails but think might be even more of a problem now. |
…te, abort the website archiving (#12708) * If one of the period archiving fails, we now cancel the remaining archiving processes * Increment the skipped counter
…ly successful (#12720) + Simplify & fix some logic
I found another use case where one can trigger the same job in parallel twice, when (For example I just had an issue while re-processing some data after it was manually invalidated, and the core:archive started again to run while my force-idsites was running) |
|
||
$periodInProgress = $this->isAlreadyArchivingAnyLowerPeriod($idSite, $period); | ||
if ($periodInProgress) { | ||
$this->logger->info("- skipping archiving for period '{period}' because processing the period '{periodcheck}' is already in progress.", array('period' => $period, 'periodcheck' => $periodInProgress)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fyi @mattab saw it in logs
skipping archiving for period 'year' because processing the period 'day' is already in progress.
* Do not run the same archive command twice * simplify * During core:archive when a day or period archiving fails for a website, abort the website archiving (matomo-org#12708) * If one of the period archiving fails, we now cancel the remaining archiving processes * Increment the skipped counter * also exit when a day period is already active * undo last commit * if day archive fails, then do not proceed archiving periods * Minor changes * Archive: mark the daily archive as successful, only after it was really successful (matomo-org#12720) + Simplify & fix some logic * do not archive a period when a sub period is processing (matomo-org#12719)
Currently, the cron archiver might start the same archiving command twice. For example when a site takes long to archive. To avoid running the same archiving process in parallel, we should check if such a job is already running and if so, not start the same archive command again.
I did some simple testing on my local machine but might need some more testing.
Only supports CLI archiving, not web archiving.