Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add possibility to run multiple archiver in parallel #4903

Closed
tsteur opened this issue Mar 25, 2014 · 25 comments
Closed

Add possibility to run multiple archiver in parallel #4903

tsteur opened this issue Mar 25, 2014 · 25 comments
Assignees
Labels
Critical Indicates the severity of an issue is very critical and the issue has a very high priority. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Milestone

Comments

@tsteur
Copy link
Member

tsteur commented Mar 25, 2014

Add possibility to run multiple archiver via cli in parallel to make archiving of >10k websites faster.

If a server has enough power / resources a user will be able to start for instance 5 archiver in parallel and they will all archive different site ids. Currently, all started archiver would archive the same site ids.

@tsteur
Copy link
Member Author

tsteur commented Mar 25, 2014

In 5fcbc09: refs #4903 started to work on the possibility to run multiple archivers in parallel for faster archiving. There were multiple issues, for instance there were arrays of siteIds read and written in Options but options do cache all values in a class property so an update of an option does not get updated on another running archiver. Also all sites were reprocessed because of the time_before_today_archive_considered_outdated setting if the last archiving by another archivier was 10 seconds or longer ago. To prevent this only maintaining a list of to be processed siteids in db / filesystem helps so far

@tsteur
Copy link
Member Author

tsteur commented Mar 25, 2014

In 3827424: refs #4903 added some comments and better naming

@hpvd
Copy link

hpvd commented Mar 25, 2014

Great idea!!
What Do you think about a setting where the advanced User can put in the max number of the cores/threads which are available in the server/should be used max by Piwik?

  • this may allow max performance for piwik
  • but do not slow down other tasks of the server/ the complete server...

Maybe this is not only interesting for archiving, but also for other processes in Piwik...
=> should be open a other ticket for the general task and collection of ideas for the global topic "parallelization of processes for best performance"?

@hpvd
Copy link

hpvd commented Mar 25, 2014

have opened a new global ticket for the collection of ideas:
#4905

@tsteur
Copy link
Member Author

tsteur commented Mar 25, 2014

In 08a0e53: refs #4903 added tests and fixed some issues

@tsteur
Copy link
Member Author

tsteur commented Mar 25, 2014

In 70cd91b: refs #4903 added cli command to trigger archiving

@tsteur
Copy link
Member Author

tsteur commented Mar 25, 2014

In 77d9bd0: refs #4903 sleep 25 milliseconds not microseconds

@tsteur
Copy link
Member Author

tsteur commented Mar 26, 2014

To run for instance 10 archiver in parallel one can execute the following command:

for ((n=0;n<10;n++)); do (/path/to/php /path/to/piwik/console core:archive --url=http://example.org/piwik &); done

FYI: It is not a good idea to do this in case your archiving tables are using the Myisam engine. More on this will follow

@tsteur
Copy link
Member Author

tsteur commented Mar 26, 2014

In c2b6c25: refs #4903 forward archive.php to new console command

@tsteur
Copy link
Member Author

tsteur commented Mar 26, 2014

In 42ff0ac: refs #4903 those tests are actually database tests

@tsteur
Copy link
Member Author

tsteur commented Mar 26, 2014

In 25c3a80: refs #4903 this might fix the tests

@tsteur
Copy link
Member Author

tsteur commented Mar 26, 2014

In 213b5c7: refs #4903 make archiving via web possible again, will provide a better solution later

@tsteur
Copy link
Member Author

tsteur commented Mar 26, 2014

In 7a2bf71: refs #4903 better solution for not working web archiving

@tsteur
Copy link
Member Author

tsteur commented Mar 27, 2014

In 4bbdd90: refs #4903 removed no longer needed methods, make sure logger uses correct log level in case logger instance was already created, if the test proxy is used do not go two directories upwards

@tsteur
Copy link
Member Author

tsteur commented Mar 27, 2014

In d40e217: refs #4903 log if we resume an already existing queue

@tsteur
Copy link
Member Author

tsteur commented Mar 27, 2014

In 3bfbb7b: refs #4903 forgot to commit this change

@tsteur
Copy link
Member Author

tsteur commented Mar 28, 2014

In 754cf8a: refs #4903 added test for web archiving. Added screen to log writers hoping no test fails because of this

@tsteur
Copy link
Member Author

tsteur commented Mar 28, 2014

In a0acaac: refs #4903 removing this lock as it does not work anyway. After locking the table the lock will be immediately released when inserting the new archiving id as there is a second get_lock call. According to MySQL and tests have proven it a lock is released when you execute a new GET_LOCK()

@tsteur
Copy link
Member Author

tsteur commented Apr 2, 2014

In 934caa2: refs #4903 before executing a scheduled task reschedule it to prevent multiple executions when multiple archiver are running

@mattab
Copy link
Member

mattab commented Apr 2, 2014

In 9fa3b75: Refs #4903 Improving the CronArchive output messages and fixing some issues

@mattab
Copy link
Member

mattab commented Apr 2, 2014

In 9d330e8: Refs #4903 Show "visits this week", "visits this month" etc.

@mattab
Copy link
Member

mattab commented Apr 2, 2014

In 47cd0bb: Refs #4903 Fix regression + tweaks to debug message
(thanks phpstorm for telling me the variable was unused)

@tsteur
Copy link
Member Author

tsteur commented Apr 3, 2014

Ran some tests in a setup with 2 nodes. One node running the PHP and one node running the DB. In this first test the PHP node was rather weak (2 cores, 8GB RAM) and it was not worth running more than 2 workers. There was no performance improvement when running 3 or 4 workers. Here it took about 40hours to archive 4k websites with 1 worker and 20 hours with 2 workers (estimated).

After running everything (PHP+DB) on the powerful DB node (16 cores, 64GB RAM) this changed. It was easily possible to run 12 workers (maybe even 16 or 20 would work, depends amount of data etc). Here it took about 8hours to archive 4k websites with 4 workers, 4.5hours with 8 workers, 3.5 hours with 12 workers (estimated).

More information:

--------------------

WEAK PHP, STRONG DB NODE

After 1 hour with 2 workers: 232/4005: 116 per worker
compared to with only 1 worker all the time 145/4005: => increase 160%

Added 3rd worker after 1hour with 2 workers...

After 2 hours with an additional 3rd worker:  449/4005: 72 per worker
compared to with only 1 worker all the time: 225/4005 ==> increase 200%

Added 4rd worker after 2hours

After 3 hours with an additional 4th worker:  676/4005: 57 per worker
compared to with only 1 worker all the time: 308/4005 ==> increase 220%

MySql load average: 0.43, 0.53, 0.58
PHP load average: 5.68, 5.19, 4.91

Estimated time to complete all 4k sites with only 1 worker: 39hours
Estimated time to complete all 4k sites with only 2 worker: 17hours
Estimated time to complete all 4k sites with only 3 worker: 18hours
Estimated time to complete all 4k sites with only 4 worker: 17.5hours

--------------------

PHP AND DB BOTH ON STRONG DB NODE

After 1 hour with 4 workers: 490/4005: 122.5 per worker
After 2 hours with 4 workers: 966/4005: 121 per worker

load average: 5.41, 4.81, 4.54

Estimated time to complete all 4k sites with only 4 worker: 8hours

--------------------

PHP AND DB BOTH ON STRONG DB NODE

After 1 hour with 8 workers: 910/4005: 113 per worker
After 2 hours with 8 workers: 1884/4005: 117 per worker

load average: 10.22, 10.16, 10.12

Estimated time to complete all 4k sites with only 4 worker: 4.5hours

--------------------

PHP AND DB BOTH ON STRONG DB NODE

After 1 hour with 8 workers: 1224/4005: 102 per worker
After 2 hours with 8 workers: 3020/4005: 189 per worker (smaller websites were coming)

load average: 9.97, 9.79, 9.72

Estimated time to complete all 4k sites with only 4 worker: 3.5hours
4K sites were actually finished after 2.25 hours as there were small websites at the end


@mattab
Copy link
Member

mattab commented Apr 17, 2014

Such a job well done, Thomas!

@mattab
Copy link
Member

mattab commented Apr 23, 2014

In 135624d: Fixes #5035 CliMulti can now trigger a limited number of concurrent requests.
This is set to three concurrent processes in core:archive command.
Refs #4903

@tsteur tsteur added this to the 2.2.1 - Piwik 2.2.1 milestone Jul 8, 2014
@tsteur tsteur self-assigned this Jul 8, 2014
@tsteur tsteur modified the milestones: 2.4.0 - Piwik 2.4.0, 2.2.1 - Piwik 2.2.1 Jul 11, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…le archivers in parallel for faster archiving. There were multiple issues, for instance there were arrays of siteIds read and written in Options but options do cache all values in a class property so an update of an option does not get updated on another running archiver. Also all sites were reprocessed because of the time_before_today_archive_considered_outdated setting if the last archiving by another archivier was 10 seconds or longer ago. To prevent this only maintaining a list of to be processed siteids in db / filesystem helps so far
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…er uses correct log level in case logger instance was already created, if the test proxy is used do not go two directories upwards
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…g writers hoping no test fails because of this
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…fter locking the table the lock will be immediately released when inserting the new archiving id as there is a second get_lock call. According to MySQL and tests have proven it a lock is released when you execute a new GET_LOCK()
sabl0r pushed a commit to sabl0r/piwik that referenced this issue Sep 23, 2014
…ncurrent requests.

This is set to three concurrent processes in core:archive command.
Refs matomo-org#4903
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Critical Indicates the severity of an issue is very critical and the issue has a very high priority. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc.
Projects
None yet
Development

No branches or pull requests

3 participants