Document new archiving process #16261

Eg in https://developer.matomo.org/guides/archiving ?

Wouldn’t need to be crazy detailed but at least start with a rough overview of how things work now, and go bit more into detail where needed just so we have it documented how we expect things to work. This might be interesting when debugging things or when someone is not clear or so.

Of course it be a bit like where do we start and where do we end… archive invalidation and archive deletions is already a big topic by itself… eg we could list when are archives deleted like

• When a segment is deleted
• When a newer archive exists
  etc

An archive is being invalidated when

• New tracking request comes in for a specific day
• Manually invalidated

• A newer archive for same day has been invalidated
  etc

• When an archive is deleted, it will also check for older partial done flags and delete them as well etc.
• Could also document what is a Partial DONE, what is an OK done flag.
• Also need to document how to trigger a rearchive for a specific plugin (and maybe a specific report).
• That we search for all archives when trying to find a report, etc.
• The new archive_invalidations table and when something is written in there and being processed (eg the flags)
• any differences now between browser and cli archiving?
• etc

That would be valuable. Also important for future new team members to get their head around it :+1:

Here is a proposal for the new sections:

1. browser archiving vs pre-archiving: basics of both approaches, some of this info is already there. for all following sections we talk about the topic for how it applies to both browser archiving and pre-archiving.
2. archive initiation: how is archiving started? answer for browser archiving & for pre-archiving
3. archive invalidation: how are existing archives invalidated, what makes an archive invalidated, how is this done in code and handled by browser vs pre-archiving
4. archive purging: how are old archives deleted? at what other times are archives deleted?
5. partial archives: what are they, how do they fit into initiation/invalidation/purging workflow, how do archivers handle this, how do plugins initiate it, + a note that it doesn't work for browser archiving

does this work?

sounds good @diosmosis

@diosmosis can we close this issue?

There are a couple changes to be made in the closed developer-documentation PR, will finish them soon.