@mattab opened this Issue on September 5th 2018 Member

Challenge: make archiving faster when there are hundreds of websites

Solution:

  • We only want to trigger the archiving for sites, that had at least one tracking request since the last archiving run. This will only help if one has many sites with 0 visits. It does not help for many sites with low traffic.
  • Applies to any archive for today / this week / this month / this year, or a custom date range that includes today.

This should probably be implemented in two places.

  • in core archiver, while preparing to create the archive, whenever there is no visit since the last archive timestamp, we should realise the last processed temporary archive is still valid, and skip creating a new one
  • in core:archive we would also be smart and automatically skip websites, and all segments, whenever there was no new visit. (we would need to skip websites and all segments only when we are sure that they were all successfully processed.)

from https://github.com/matomo-org/matomo/issues/5922#issuecomment-90352741

@diosmosis commented on February 12th 2019 Member

@mattab / @tsteur:

  • in core:archive we would also be smart and automatically skip websites, and all segments, whenever there was no new visit.

Would this cause problems for plugins that force archiving even if no visits?

@mattab commented on February 12th 2019 Member
  • in core:archive we would also be smart and automatically skip websites, and all segments, whenever there was no new visit.

Would this cause problems for plugins that force archiving even if no visits?

when plugins force archiving in this way, they would have good reasons to do so, so their forcing would overwrite any logic we add around skipping websites/segments/etc

@diosmosis commented on February 12th 2019 Member

@mattab I think the first bullet point actually solves that as well, doesn't it? Ie, if the archiver sends requests for segments, we'll see there are no visits and avoid archiving. Adding this logic to CronArchive.php while keeping some logic in PluginsArchiver.php for plugins that force archiving would be rather difficult I think.

@diosmosis commented on February 18th 2019 Member

@mattab can you reply to my last comment?

@mattab commented on February 18th 2019 Member

The first bullet point (core archiver) fixes part of the problem, which may be enough for now, but thought there would be a lot of improvements hidden in the other second bullet point (core:archive / CronArchive). for example imagine a Matomo with 1,000 sites and 10 global segments so 10,000 segments. If we only do the first bullet point, we still need to send 10,000 requests * 5 periods = 50K requests.. which would take a long time, possibly hours? what do you think (not sure if my numbers are correct)?

@diosmosis commented on February 18th 2019 Member

The numbers seem right, but since we don't know if a plugin will force archiving, we can't really skip those requests... not w/o taking the forcing outside of the core archiving logic and putting it in CronArchive. I think that would be rather non-trivial.

@diosmosis commented on February 23rd 2019 Member

Maybe @tsteur has an idea or some thoughts?

@tsteur commented on February 24th 2019 Member

maybe it could be implemented only when type != rollup? or not sure if other plugins / website types force it...

Ideally otherwise we just set a flag for each site when starting or finished archiving like option.hadVisitsSinceLastArchiving= false and as soon as there is a request, we could set option.hadVisitSinceLastArchiving= true. Don't think we have such a flag currently as visits may be imported in the past etc.

Can you otherwise mention some plugins that force archiving? could they listen/reuse the same event in CronArchive? Would need to look how those plugins do it & why

@diosmosis commented on February 24th 2019 Member

@tsteur I think the event that needs to be respected is Archiving.getIdSitesToArchiveWhenNoVisits. I was able to implement the first bullet in a PR, but the second seems harder since we would want to avoid sending requests in CronArchive when there are no visits. However, looking at the event, maybe we can post it in CronArchive too...

@tsteur commented on February 24th 2019 Member

Posting it again may help 👍 Not sure for what we added this event?

@diosmosis commented on February 24th 2019 Member

I don't think I was here for it, but I thought it was for something like SEO or importing stats from somewhere else? Where we'd want to archive, but not be dependent on visits in Matomo.

@tsteur commented on February 25th 2019 Member

No idea, I can't see it in any of our plugins... also can't find it in an issue or changelog

@diosmosis commented on March 1st 2019 Member

@mattab / @tsteur implemented the change in the code and think again it might be a pointless addition. Looks like it's already in CronArchive.php: https://github.com/matomo-org/matomo/blob/3.x-dev/core/CronArchive.php#L1261-L1292

This code will check if there are visits between the last successful archiving time, which is pretty close to the latest archive for today. I think the only case it will save time is if archiving fails, but some archives get created. What do you think, still useful? You can see my changes here: https://github.com/matomo-org/matomo/pull/14091/commits/23b5a1871f3c332b4ebc46235814a914abc3e7af

Powered by GitHub Issue Mirror