Set default limit for concurrent archivers #18375
Comments
Starker3
added
the
Enhancement
|
BTW another solution, without possibly breaking it could be to simply adjust the guide. There will be potentially always few people where it would break something, just the same way as it would be fixing something for few potentially. We could maybe then change the default in Matomo 5 or so. |
|
Sounds good to maybe include this sort of change in a bigger version release. Changing the guide will help people who set it up in the future, but I think by setting a default like this could potentially help quite a few people who already have it setup and maybe don't even realise that sometimes their server is slow for this reason. |
|
This could actually be something that is easily setup if we work on #17719 |
|
Will do now as we aren't likely to work on #17719 for a while. |
|
So what should we change as part of this issue? Only change the default limit of concurrent archivers to 3. Or also an additional option to force running an archiving even if the limit is already hit? |
|
We would
|
|
@tsteur Did a simple PR for that one, trying to complete those two. But looks really simple, am I on the right path?
|
|
@peterhashair in this case it be more about the archive command https://github.com/matomo-org/matomo/blob/4.12.3/plugins/CoreConsole/Commands/CoreArchiver.php#L114 which people set up as part of https://matomo.org/docs/setup-auto-archiving/ The value set in the PR would be otherwise overwritten in https://github.com/matomo-org/matomo/blob/4.12.3/plugins/CoreConsole/Commands/CoreArchiver.php#L47 I believe. |
|
we need to update this https://developer.matomo.org/guides/archiving-behavior-specification once |
|
This should be ready to action, if you haven't already @peterhashair ? |
|
@justinvelluppillai updated in here matomo-org/developer-documentation#680 |
Currently when someone sets up a core:archive crontab, the recommendation is to just simply set it without any additional options: https://matomo.org/docs/setup-auto-archiving/
In a lot of cases this works without any issues. However, if for some reason a particular run of the core:archive takes longer than usual (For example lots of tracking data all of a sudden) or for instances that track a lot data already this may lead to many concurrent archivers running at the same time.
This in turn results in even slower archiving performance and becomes a negative feedback loop eventually resulting in the database server hitting 100% CPU usage.
It would be great if we could set a default limit to the number of concurrently running archivers (Maybe some like a default max of 3 archivers).
I don't think this would have any impact on people not already setting the number of concurrent archivers. The one negative would be that if someone wanted to run a manual core:archive command and there are already core:archive commands running, then this would cause that archiver to exit.
In this case maybe an additional option to disable the check for maximum archivers might work for example. (We could also hint this in the output when the archiver exits for this reason).
One example of how this impacted 2 customers recently: One customer ended up with over 60 core:archive crontabs starting over a period of one or two days. This caused the CPU of the Reader database to hit 100% consistently and none of the core:archivers were able to finish because the database server was already way overloaded with requests.
The other customer did not have a Reader database setup and they ended up with around 25 concurrent archivers running which caused CPU to hit 100% consistently. This prevented them from being able to use Matomo (They were not even able to load the UI)
The text was updated successfully, but these errors were encountered: