Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document new archiving process #16261

Closed
tsteur opened this issue Aug 2, 2020 · 5 comments
Closed

Document new archiving process #16261

tsteur opened this issue Aug 2, 2020 · 5 comments
Assignees
Labels
c: Documentation For issues related to in-app product help messages, or to the Matomo knowledge base. c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself.
Milestone

Comments

@tsteur
Copy link
Member

tsteur commented Aug 2, 2020

Eg in https://developer.matomo.org/guides/archiving ?

Wouldn’t need to be crazy detailed but at least start with a rough overview of how things work now, and go bit more into detail where needed just so we have it documented how we expect things to work. This might be interesting when debugging things or when someone is not clear or so.

Of course it be a bit like where do we start and where do we end… archive invalidation and archive deletions is already a big topic by itself… eg we could list when are archives deleted like

  • When a segment is deleted
  • When a newer archive exists
    etc

An archive is being invalidated when

  • New tracking request comes in for a specific day

  • Manually invalidated

  • A newer archive for same day has been invalidated
    etc

  • When an archive is deleted, it will also check for older partial done flags and delete them as well etc.

  • Could also document what is a Partial DONE, what is an OK done flag.

  • Also need to document how to trigger a rearchive for a specific plugin (and maybe a specific report).

  • That we search for all archives when trying to find a report, etc.

  • The new archive_invalidations table and when something is written in there and being processed (eg the flags)

  • any differences now between browser and cli archiving?

  • etc

@tsteur tsteur added the c: Documentation For issues related to in-app product help messages, or to the Matomo knowledge base. label Aug 2, 2020
@tsteur tsteur added this to the 4.0.0 milestone Aug 2, 2020
@mattab
Copy link
Member

mattab commented Aug 4, 2020

That would be valuable. Also important for future new team members to get their head around it 👍

@diosmosis
Copy link
Member

Here is a proposal for the new sections:

  1. browser archiving vs pre-archiving: basics of both approaches, some of this info is already there. for all following sections we talk about the topic for how it applies to both browser archiving and pre-archiving.
  2. archive initiation: how is archiving started? answer for browser archiving & for pre-archiving
  3. archive invalidation: how are existing archives invalidated, what makes an archive invalidated, how is this done in code and handled by browser vs pre-archiving
  4. archive purging: how are old archives deleted? at what other times are archives deleted?
  5. partial archives: what are they, how do they fit into initiation/invalidation/purging workflow, how do archivers handle this, how do plugins initiate it, + a note that it doesn't work for browser archiving

does this work?

@tsteur
Copy link
Member Author

tsteur commented Aug 4, 2020

sounds good @diosmosis

@tsteur
Copy link
Member Author

tsteur commented Aug 19, 2020

@diosmosis can we close this issue?

@diosmosis
Copy link
Member

There are a couple changes to be made in the closed developer-documentation PR, will finish them soon.

@mattab mattab added the c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. label Sep 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Documentation For issues related to in-app product help messages, or to the Matomo knowledge base. c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself.
Projects
None yet
Development

No branches or pull requests

3 participants