Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallelization of processes for best performance #4905

Closed
hpvd opened this issue Mar 25, 2014 · 8 comments
Closed

Parallelization of processes for best performance #4905

hpvd opened this issue Mar 25, 2014 · 8 comments
Labels
c: Performance For when we could improve the performance / speed of Matomo. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.

Comments

@hpvd
Copy link

hpvd commented Mar 25, 2014

For best speed on todays multicore/mulithread servers,
piwik should be able to run as many processes as possible in parallel mode.

But which are the processes which can run in parallel mode??

This ticket is about a collection of ideas, and can be divided into several single tickets for each upcoming idea.

it was inspired from ticket:
#4903

One side question which has to be answered:
how to tell piwik how many cores/threads should be used max?

What Do you think about a setting where the advanced User can put in the max number of the cores/threads which are available in the server/should be used max. by Piwik?

  • this may allow max performance for piwik
  • but do not slow down other tasks of the server/ the complete server...
@hpvd
Copy link
Author

hpvd commented Mar 25, 2014

  1. archiver in parallel (see Add possibility to run multiple archiver in parallel #4903)

@hpvd
Copy link
Author

hpvd commented Mar 25, 2014

  1. activate multicore support but also core limitation (see ticket) by default for image processing (image magic)

this has another goal but the details/code maybe interesting: http://www.daniloaz.com/en/617/systems/high-cpu-load-when-converting-images-with-imagemagick/

@hpvd
Copy link
Author

hpvd commented Mar 25, 2014

  1. activate multicore support but also core/thread limitation (see ticket) by default for zip processing

@hpvd
Copy link
Author

hpvd commented Mar 25, 2014

maybe one should modify the title of this ticket to:
"Controlled Parallelization of processes for best performance"
because it should not only be possible to minify the time used by an action by maximized parallelization
but it is also very important to be able to control the load applied by piwik to the sever by limiting the used cores/threads

@mattab
Copy link
Member

mattab commented Mar 25, 2014

Thanks for the report, but I think this ticket is too broad.
so far we are only thinking of the archiver.

What parts do you mean we can parallelize?

@hpvd
Copy link
Author

hpvd commented Mar 27, 2014

hmm maybe you can have a look in the ticket text again:
"This ticket is about a collection of ideas, and can be divided into several single tickets for each upcoming idea."

=> The first concrete ideas can already be found within the comments.

If theses approaches are fine, I or everyone else can add single tickets for each and make a link to these ticket within these ticket comments

And when there are one day

  • no more ideas regarding the topic Parallelization / Multicore utilization and
  • each approach has his own ticket
    then it's time to close this ticket :-)
    What do you think?

@mattab
Copy link
Member

mattab commented Mar 27, 2014

I should have left it open as it's an interesting discussion!
cheers for your active participation here also.

@hpvd hpvd added this to the Future releases milestone Jul 8, 2014
@mattab mattab removed the P: normal label Aug 3, 2014
@mattab mattab modified the milestones: Long term, Mid term Dec 23, 2015
@mattab mattab modified the milestones: Long term, Mid term Dec 5, 2016
@mattab
Copy link
Member

mattab commented Aug 13, 2019

Now core:archive can be parallelised for already a few years 🚀
the parameters you can use are:

 --concurrent-requests-per-website  When processing a website and its segments, number of requests to process in parallel (default: 3)
 --concurrent-archivers             The number of max archivers to run in parallel. Depending on how you start the archiver as a cronjob, you may need to double the amount of archivers allowed if the same process appears twice in the `ps ex` output.

see ./console help core:archive or: https://matomo.org/docs/setup-auto-archiving/

@mattab mattab closed this as completed Aug 13, 2019
@mattab mattab added the not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. label Aug 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Projects
None yet
Development

No branches or pull requests

2 participants