Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

5363 cron archive full concurrency #6755

Closed
wants to merge 83 commits into from

Conversation

diosmosis
Copy link
Member

Refs #5363, make CronArchive algorithm fully concurrent by adding all API requests made to queue and executing requests in the queue in parallel. Includes:

  • Job queue/processor interfaces with default implementation. Default queue/processor uses MySQL and CliMulti only, so should work in all supported environments. Code allows using custom queues/processors that use other technologies to be developed and used however.
    • Default job queue uses a 'jobs' table. It is created on demand and it is possible to use multiple tables for it to have multiple queues w/ different jobs (eg, low priority jobs in one queue + high priority jobs in main queue).
    • Default job processor pulls a number of jobs from the queue and executes them using CliMulti to achieve parallelism. Job hook logic is always executed in the same thread as the job processor, not the job itself.
  • DI integration. Job queues & processors can be defined in DI config and referenced by name to core:archive + core:jobs-server commands.
  • Cleaner CronArchive code/architecture. Uses aspect oriented programming in Piwik\CronArchive\Hooks class to separate core algorithm logic with logging, statistics & options. Algorithm logic is stored in AlgorithmRules class and is cached so concurrent code does not re-compute everything (because new CronArchive instances currently have to be created in jobFinished/Starting hooks and of course also on different machines). Algorithm output is also cleaner.
  • New core:jobs-server command that can be run on extra machines to have other machines process jobs.
  • Jobs are stored serialized in queues and unserialized via a new method: Piwik::secureUnserialize.
  • An API method CoreAdminHome.executeJob allows Jobs to be defined that do not use URLs. The CliProcessor will use a URL to this API method to execute such jobs via CliMulti. The API method is only necessary for CliProcessor. Other processors should not need it.

Still needs more testing and I haven't reviewed everything yet.

diosmosis added 30 commits September 21, 2014 19:29
…ncy primitive and converted big chunks of CronArchive to use job/consumer. Also moved a lot of logic into new SiteArchivingInfo class.
Conflicts:
	core/CliMulti.php
	core/CronArchive.php
… class, rename SiteArchivingInfo to AlgorithmState and move stat fields to AlgorithmStatistics.
…hmState. Removed function CronArchive::initStateFromParameters.
@mattab mattab added Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. c: Performance For when we could improve the performance / speed of Matomo. c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. labels Dec 1, 2014
@mattab mattab added this to the Piwik 2.10.0 milestone Dec 1, 2014
@mattab mattab modified the milestones: Piwik 2.11.0, Piwik 2.10.0 Dec 6, 2014
@mattab
Copy link
Member

mattab commented Dec 7, 2014

Postponed until further notice. See: #5363 (comment)

@mattab mattab modified the milestones: Short term, Piwik 2.11.0 Dec 7, 2014
@mattab mattab closed this Dec 10, 2014
@mattab mattab added the wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it. label Dec 18, 2014
@mattab mattab deleted the 5363_cron_archive_full_concurrency branch September 22, 2015 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo. c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. Enhancement For new feature suggestions that enhance Matomo's capabilities or add a new report, new API etc. wontfix If you can reproduce this issue, please reopen the issue or create a new one describing it.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants