Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create temporary table for segments only once instead of twice when archiving #14818

Closed
tsteur opened this issue Aug 26, 2019 · 0 comments · Fixed by #14788
Closed

Create temporary table for segments only once instead of twice when archiving #14818

tsteur opened this issue Aug 26, 2019 · 0 comments · Fixed by #14788
Assignees
Labels
c: Performance For when we could improve the performance / speed of Matomo. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Milestone

Comments

@tsteur
Copy link
Member

tsteur commented Aug 26, 2019

Currently, when we archive, we initialize the PluginsArchiver twice. Once for core and once for the plugins here: https://github.com/matomo-org/matomo/blob/3.12.0-b2/core/ArchiveProcessor/Loader.php#L101-L117

In the PluginsArchiver's constructor we then first cleanup any possible temp table that already exists ( https://github.com/matomo-org/matomo/blob/3.12.0-b2/core/ArchiveProcessor/PluginsArchiver.php#L70-L71) and then later create the temp table as soon as a LogAggregator::generateQuery is called.

This means what we do is:

  • Start archiving core metrics

  • Cleanup temp table for segment if for some reason was not removed (eg a fatal error or memory issue etc and old MySQL version is used)

  • Create temp table for segment

  • archive core metrics

  • Finalise archive

  • Cleanup temp table for segment ===> This is the step we want to prevent but only if we actually go into the plugins archiver again to archive plugin metrics

  • Start archiving plugins metrics

  • Cleanup temp table for segment if for some reason was not removed (eg a fatal error or memory issue etc and old MySQL version is used) ===> This is the step we want to prevent

  • archive plugin metrics

  • Finalise archive

  • Cleanup temp table for segment

This will make segmenting again a bit faster as we avoid running the expensive segment query and avoid lots of inserts...

@tsteur tsteur added c: Performance For when we could improve the performance / speed of Matomo. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. labels Aug 26, 2019
@tsteur tsteur self-assigned this Aug 26, 2019
@mattab mattab added this to the 3.12.0 milestone Oct 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants