@diosmosis opened this Pull Request on February 1st 2020 Member

New workflow

The current workflow for core:archive is to check if there are visits for a site, then launching archiving w/ lastN days/weeks/months/years. The new workflow is: invalidate archive, run core:archive and let it process the queue.

Relevant changes

  • Archive invalidation now creates archive rows w/ NULL ts_archived if there isn't an existing archive to set the done value for. This allows core:archive to notice and process the archive, even though one never existed beforehand.
  • core:archive now selects the next invalidated archive from the archive tables and marks it as in progress so we don't have to look for commands that are already running to know if an archive is being processed.

Notes

  • there are some bug fixes to the archive invalidation code here, some archives were not invalidated that should be, and some archives were invalidated that did not need to be.
  • there is an optimization in Loader.php that vastly improves the performance for archiving sites with no visits (as demonstration, all the system test jobs are much faster now; the one w/ ArchiveCronTest eg has improved from 21 mins to 6 mins).

Fixes #15117

@diosmosis commented on March 30th 2020 Member

@tsteur this PR is ready for an initial review. tests might still need some fixing (haven't checked the last build since I dealt w/ some TODOs). There are some TODOs in the code, those are things I wasn't sure how to solve completely myself, so would appreciate your opinion.

Powered by GitHub Issue Mirror