Concurrently running archivers can trigger scheduled tasks multiple times #19287

When users have a high number of sites or a lot of data, it can be expected to have two (Or more) concurrently running core:archive tasks that could take several hours to all complete.

In the case where a user has a high number of sites where each site is relatively low volume or a high number of sites where each site takes roughly the same amount of time to complete we can run into a situation where two concurrently running archivers finish at exactly the same time.

For example in this case where a user had two concurrently running core:archive tasks, one started an hour after the other. We can see here that they both finished at roughly the same time:

INFO [2022-05-15 00:29:43] 3017001  Done archiving!
INFO [2022-05-15 00:29:43] 3017001  ---------------------------
INFO [2022-05-15 00:29:43] 3017001  SUMMARY
INFO [2022-05-15 00:29:43] 3017001  Processed 2049 archives.
INFO [2022-05-15 00:29:43] 3017001  Total API requests: 2049
INFO [2022-05-15 00:29:43] 3017001  done: 2049 req, 3101515 ms, no error
INFO [2022-05-15 00:29:43] 3017001  Time elapsed: 3101.515s
INFO [2022-05-15 00:29:43] 3017001  ---------------------------
INFO [2022-05-15 00:29:43] 3017001  SCHEDULED TASKS
INFO [2022-05-15 00:29:43] 3017001  Starting Scheduled tasks... 

INFO [2022-05-15 00:29:46] 2897212  Done archiving!
INFO [2022-05-15 00:29:46] 2897212  ---------------------------
INFO [2022-05-15 00:29:46] 2897212  SUMMARY
INFO [2022-05-15 00:29:46] 2897212  Processed 4258 archives.
INFO [2022-05-15 00:29:46] 2897212  Total API requests: 4258
INFO [2022-05-15 00:29:46] 2897212  done: 4258 req, 6704938 ms, no error
INFO [2022-05-15 00:29:46] 2897212  Time elapsed: 6704.938s
INFO [2022-05-15 00:29:46] 2897212  ---------------------------
INFO [2022-05-15 00:29:46] 2897212  SCHEDULED TASKS

We can then end up with a situation where they execute the exact same scheduled tasks at the same exact time causing things such as duplicate email reports to be sent:

INFO [2022-05-15 00:33:32] 2897212  Scheduler: finished. Time elapsed: 225.650s
INFO [2022-05-15 00:33:32] 3017001  Scheduler: finished. Time elapsed: 228.735s
INFO [2022-05-15 00:33:32] 3017001  Scheduler: executing task Piwik\Plugins\Login\Tasks.cleanupBruteForceLogs...
INFO [2022-05-15 00:33:32] 2897212  Scheduler: executing task Piwik\Plugins\Login\Tasks.cleanupBruteForceLogs...
INFO [2022-05-15 00:33:32] 2897212  Scheduler: finished. Time elapsed: 0.004s
INFO [2022-05-15 00:33:32] 3017001  Scheduler: finished. Time elapsed: 0.004s
INFO [2022-05-15 00:33:32] 2897212  Scheduler: executing task Piwik\Plugins\TwoFactorAuth\Tasks.cleanupTwoFaCodesUsedRecently...
INFO [2022-05-15 00:33:32] 3017001  Scheduler: executing task Piwik\Plugins\TwoFactorAuth\Tasks.cleanupTwoFaCodesUsedRecently...
INFO [2022-05-15 00:33:32] 2897212  Scheduler: finished. Time elapsed: 0.001s
INFO [2022-05-15 00:33:32] 3017001  Scheduler: finished. Time elapsed: 0.001s
INFO [2022-05-15 00:33:32] 2897212  Scheduler: executing task Piwik\Plugins\UsersManager\Tasks.cleanupExpiredTokens...
INFO [2022-05-15 00:33:32] 3017001  Scheduler: executing task Piwik\Plugins\UsersManager\Tasks.cleanupExpiredTokens...
INFO [2022-05-15 00:33:32] 2897212  Scheduler: finished. Time elapsed: 0.022s
INFO [2022-05-15 00:33:32] 3017001  Scheduler: finished. Time elapsed: 0.024s
INFO [2022-05-15 00:33:32] 2897212  Scheduler: executing task Piwik\Plugins\UsersManager\Tasks.setUserDefaultReportPreference...
INFO [2022-05-15 00:33:32] 3017001  Scheduler: executing task Piwik\Plugins\UsersManager\Tasks.setUserDefaultReportPreference...
INFO [2022-05-15 00:33:32] 3017001  Scheduler: finished. Time elapsed: 0.135s
INFO [2022-05-15 00:33:32] 3017001  Scheduler: executing task Piwik\Plugins\ScheduledReports\API.sendReport_3...
INFO [2022-05-15 00:33:33] 2897212  Scheduler: finished. Time elapsed: 0.149s
INFO [2022-05-15 00:33:33] 2897212  Scheduler: executing task Piwik\Plugins\ScheduledReports\API.sendReport_3...