Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segments may not be archived anymore when there are archive invalidations for segments that don't exist #19892

Open
tsteur opened this issue Oct 20, 2022 · 1 comment
Labels
Bug For errors / faults / flaws / inconsistencies etc.

Comments

@tsteur
Copy link
Member

tsteur commented Oct 20, 2022

see PG-820

When there are too many invalid archive invalidations, then the archiving may stop archive any segment data.

There was a bug in Roll-Up Reporting where it would create archive invalidations that the archiver cannot process and considers invalid. In particular it goes into: https://github.com/matomo-org/matomo/blob/4.12.0-rc1/core/CronArchive/QueueConsumer.php#L539-L542

In matomo/QueueConsumer.php at 4.12.0-rc1 · matomo-org/matomo where it finds the next invalidated archive there is a limit of 100 tries to find an archive to process. However, there are currently for example around 14256 invalidated archives for idsite 13 that it cannot process because there is no matching segment. Meaning after the first 100 out of these 14256 it always stops and never proceeds to the one it actually can and should process.

Maybe, in matomo/QueueConsumer.php at 4.12.0-rc1 · matomo-org/matomo we also need to delete the archive invalidation similar to how we do this in matomo/QueueConsumer.php at 4.12.0-rc1 · matomo-org/matomo. I’m not sure if there’s a reason we’re not doing it or not.

To reproduce (I haven't actually checked it can be reproduced like this):

  • Have some valid archive invalidations for existing segments for today
  • Create more than 100 invalid archive invalidations for segments that don't exist for today
  • Call ./console core:archive
  • Depending of the order it may not archive any of these segments.

Another way to reproduce this is described in PG-820 via the RollUp Reporting plugin (it's fixed in latest version)

@tsteur tsteur added the Bug For errors / faults / flaws / inconsistencies etc. label Oct 20, 2022
@bx80
Copy link
Contributor

bx80 commented Oct 26, 2022

Hi @tsteur, thanks for reporting this in detail, seems like we should always be deleting invalidations once they've served their purpose. I've assigned this issue for prioritization, but please add the 'needs priority review' label if you think it needs more urgency.

@bx80 bx80 added this to the For Prioritization milestone Oct 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc.
Projects
None yet
Development

No branches or pull requests

2 participants