For best speed on todays multicore/mulithread servers,
piwik should be able to run as many processes as possible in parallel mode.
But which are the processes which can run in parallel mode??
This ticket is about a collection of ideas, and can be divided into several single tickets for each upcoming idea.
it was inspired from ticket:
One side question which has to be answered:
how to tell piwik how many cores/threads should be used max?
What Do you think about a setting where the advanced User can put in the max number of the cores/threads which are available in the server/should be used max. by Piwik?
2) activate multicore support but also core limitation (see ticket) by default for image processing (image magic)
this has another goal but the details/code maybe interesting: http://www.daniloaz.com/en/617/systems/high-cpu-load-when-converting-images-with-imagemagick/
3) activate multicore support but also core/thread limitation (see ticket) by default for zip processing
maybe one should modify the title of this ticket to:
"Controlled Parallelization of processes for best performance"
because it should not only be possible to minify the time used by an action by maximized parallelization
but it is also very important to be able to control the load applied by piwik to the sever by limiting the used cores/threads
Thanks for the report, but I think this ticket is too broad.
so far we are only thinking of the archiver.
What parts do you mean we can parallelize?
hmm maybe you can have a look in the ticket text again:
"This ticket is about a collection of ideas, and can be divided into several single tickets for each upcoming idea."
=> The first concrete ideas can already be found within the comments.
If theses approaches are fine, I or everyone else can add single tickets for each and make a link to these ticket within these ticket comments
And when there are one day
I should have left it open as it's an interesting discussion!
cheers for your active participation here also.
core:archive can be parallelised for already a few years :rocket:
the parameters you can use are:
--concurrent-requests-per-website When processing a website and its segments, number of requests to process in parallel (default: 3) --concurrent-archivers The number of max archivers to run in parallel. Depending on how you start the archiver as a cronjob, you may need to double the amount of archivers allowed if the same process appears twice in the `ps ex` output.
./console help core:archive or: https://matomo.org/docs/setup-auto-archiving/