@tsteur opened this Issue on October 20th 2022 Member

see PG-820

When there are too many invalid archive invalidations, then the archiving may stop archive any segment data.

There was a bug in Roll-Up Reporting where it would create archive invalidations that the archiver cannot process and considers invalid. In particular it goes into: https://github.com/matomo-org/matomo/blob/4.12.0-rc1/core/CronArchive/QueueConsumer.php#L539-L542

In matomo/QueueConsumer.php at 4.12.0-rc1 · matomo-org/matomo where it finds the next invalidated archive there is a limit of 100 tries to find an archive to process. However, there are currently for example around 14256 invalidated archives for idsite 13 that it cannot process because there is no matching segment. Meaning after the first 100 out of these 14256 it always stops and never proceeds to the one it actually can and should process.

Maybe, in matomo/QueueConsumer.php at 4.12.0-rc1 · matomo-org/matomo we also need to delete the archive invalidation similar to how we do this in matomo/QueueConsumer.php at 4.12.0-rc1 · matomo-org/matomo. I’m not sure if there’s a reason we’re not doing it or not.

To reproduce (I haven't actually checked it can be reproduced like this):

  • Have some valid archive invalidations for existing segments for today
  • Create more than 100 invalid archive invalidations for segments that don't exist for today
  • Call ./console core:archive
  • Depending of the order it may not archive any of these segments.

Another way to reproduce this is described in PG-820 via the RollUp Reporting plugin (it's fixed in latest version)

@bx80 commented on October 26th 2022 Contributor

Hi @tsteur, thanks for reporting this in detail, seems like we should always be deleting invalidations once they've served their purpose. I've assigned this issue for prioritization, but please add the 'needs priority review' label if you think it needs more urgency.

Powered by GitHub Issue Mirror