Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

archive.sh could execute in multithreaded mode for better performance #2563

Closed
mattab opened this issue Jul 12, 2011 · 5 comments
Closed

archive.sh could execute in multithreaded mode for better performance #2563

mattab opened this issue Jul 12, 2011 · 5 comments
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it.
Milestone

Comments

@mattab
Copy link
Member

mattab commented Jul 12, 2011

We can safely multithread archive.sh script, by running different website ID processing on multiple cores. We can not run archiving for a same website on multiple core, but it will improve performance to run different website ID processing at the same time.

It will improve performance especially in the case of a Piwik install with hundreds/thousands of small/medium websites.

@cbay
Copy link
Contributor

cbay commented Jul 12, 2011

Attachment:
archive.multithreaded.sh.diff

@cbay
Copy link
Contributor

cbay commented Jul 12, 2011

Here is a first version against trunk. Not compatible with #2440, and I'm not sure it's possible to have both features at the same time, at least with the approach I took (using xargs).

@mattab
Copy link
Member Author

mattab commented Jul 15, 2011

See also #2327

@julienmoumne
Copy link
Member

I've read in several tickets related to this one that multi-threading should not be activated on mono core.

Please correct me if I'm wrong but during archiving there is a lot of communication done with the database. As far as I understand scheduling, the scheduler is supposed to switch threads when they are waiting on resources (ie. network and database). During that time, threads needing CPU time (ie. for calculations and aggregations) are given CPU time.

Does it logically follow that activating multi-threading on a single core would increase performance in case the web server and the database are located on different hardware ?

@mattab
Copy link
Member Author

mattab commented Sep 13, 2011

I think considering the huge work done on the new archive.php, we should completely deprecate archive.sh and .ps1 and change all doc to use archive.php and only maintain this one.

@mattab mattab added this to the 1.6 Piwik 1.6 milestone Jul 8, 2014
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it.
Projects
None yet
Development

No branches or pull requests

3 participants