@mattab opened this Issue on September 5th 2018 Member

Challenge: make archiving faster when there are hundreds of websites

Solution:

  • We only want to trigger the archiving for sites, that had at least one tracking request since the last archiving run. This will only help if one has many sites with 0 visits. It does not help for many sites with low traffic.
  • Applies to any archive for today / this week / this month / this year, or a custom date range that includes today.

This should probably be implemented in two places.

  • in core archiver, while preparing to create the archive, whenever there is no visit since the last archive timestamp, we should realise the last processed temporary archive is still valid, and skip creating a new one
  • in core:archive we would also be smart and automatically skip websites, and all segments, whenever there was no new visit. (we would need to skip websites and all segments only when we are sure that they were all successfully processed.)

from https://github.com/matomo-org/matomo/issues/5922#issuecomment-90352741

Powered by GitHub Issue Mirror