Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tweak behaviour of orphaned segments #14828

Closed
tsteur opened this issue Aug 28, 2019 · 0 comments · Fixed by #14857
Closed

Tweak behaviour of orphaned segments #14828

tsteur opened this issue Aug 28, 2019 · 0 comments · Fixed by #14857
Assignees
Labels
c: Performance For when we could improve the performance / speed of Matomo. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone

Comments

@tsteur
Copy link
Member

tsteur commented Aug 28, 2019

Weekly we have a cronjob to delete orphaned segments in CoreAdminHome\Tasks. Meaning we delete all segment archives of segments that no longer exist.

This behaviour currently works like this:

  • Get a list of all segments
  • Iterate over each archive table
    • Get the idArchives of archives whose done flag is not in the list of existing segments
    • Delete the idArchives

Now that this command ran a few times and we have cleaned up all segment archives, and now that we implement a proper soft delete in #14826 we could make this behaviour more efficient:

  • Get all segments where deleted = 1 and ts_last_edit >= 1 week ago (assuming we run the task weekly). These are the segments that were deleted last week
  • Iterate over all archive tables
    • Get the idarchives of segments having the done flag for the deleted segment
    • Delete these idarchives from numeric and blob table

I was thinking we would only need to loop over archive tables since ts_created but we need to loop likely over all archive tables since some users will have enabled to automatically archive segments for all historical data which is also the default process_new_segments_from = "beginning_of_time"

This way things will be faster as we only execute logic if a segment was deleted

@tsteur tsteur added Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. c: Performance For when we could improve the performance / speed of Matomo. labels Aug 28, 2019
@tsteur tsteur added this to the 3.12.0 milestone Aug 28, 2019
@katebutler katebutler self-assigned this Sep 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants