Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check that when a website (or a segment) is deleted, we also delete all archive data (numeric, blob) #13966

Closed
mattab opened this issue Jan 16, 2019 · 5 comments · Fixed by #14317
Closed

Check that when a website (or a segment) is deleted, we also delete all archive data (numeric, blob) #13966

mattab opened this issue Jan 16, 2019 · 5 comments · Fixed by #14317
Assignees
@katebutler
Labels
c: Performance For when we could improve the performance / speed of Matomo.
Milestone
3.10.0

Comments

@mattab
Copy link
Member

mattab commented Jan 16, 2019

In order to space disk size and costs, when a website is deleted, or when a segment is deleted, we need to make sure that archived reports stored in the archive_numeric* and archive_blob* tables are also deleted.
@mattab mattab added the c: Performance For when we could improve the performance / speed of Matomo. label Jan 16, 2019
@mattab mattab added this to the 3.10.0 milestone Jan 16, 2019
@tsteur
Copy link
Member

tsteur commented Jan 18, 2019

refs #13976 when a user is deleted, we need to make sure to also delete all segments and their archives.

@mattab
Copy link
Member Author

mattab commented Feb 20, 2019

Notes from a call:

  • likely this could be implemented as a job that deletes all archives that are not part of the "Known segments"
  • for each archive table, get the list of unique done flags for each segment known in the system
    • use this list to delete all the other archives (whose done flags are not in this list)
  • making sure only archives older than 24 hours are deleted (as newer archives may be currently in use)

@katebutler katebutler self-assigned this Mar 13, 2019
@tsteur
Copy link
Member

tsteur commented Mar 13, 2019

@mattab how about segments that aren't automatically archived. I suppose we keep them in the archive tables? or delete them as well when they are older?

@tsteur
Copy link
Member

tsteur commented Mar 13, 2019

@mattab are there maybe also range archives that we can delete?

@mattab
Copy link
Member Author

mattab commented Mar 13, 2019

Segments that aren't automatically archived, keep them forever.
Range archives might be already deleted automatically after some time, iirc. So there shouldn't need to do anything special with then.

@katebutler katebutler mentioned this issue Apr 5, 2019
Merged
@mattab mattab modified the milestones: 3.11.0, 3.10.0 May 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@katebutler katebutler

Labels
c: Performance For when we could improve the performance / speed of Matomo.
Projects
None yet
Milestone
3.10.0
Development

Successfully merging a pull request may close this issue.

Task to purge archives for deleted websites and segments matomo-org/matomo
3 participants
@tsteur @mattab @katebutler