Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Research if Mysql Partitioning can be used for better performance? #6715

Open
mattab opened this issue Nov 23, 2014 · 3 comments
Open

Research if Mysql Partitioning can be used for better performance? #6715

mattab opened this issue Nov 23, 2014 · 3 comments
Labels
c: Performance For when we could improve the performance / speed of Matomo. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.

Comments

@mattab
Copy link
Member

mattab commented Nov 23, 2014

The goal of this issue is to research whether Mysql partitioning could be used in Piwik to provide better performance.

A few years ago this was not an option because they didn't support partitioning by datetime column.
Now I see that they added the feature in Mysql 5.6. It could be helpful for partitioning tracker log_* tables by monthly partition. Sub-partitioning sounds also interesting, so maybe we could sub-partition log data by day.

@mattab mattab added Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. c: Performance For when we could improve the performance / speed of Matomo. labels Nov 23, 2014
@mattab mattab added this to the Long term milestone Nov 23, 2014
@mattab mattab modified the milestones: Long term, Mid term Dec 23, 2015
@mattab mattab modified the milestones: Long term, Mid term Dec 5, 2016
@toredash
Copy link
Contributor

Hope this information is useful;
Running Matomo with AWS RDS Serverless is for us a cost-effective way to handle our dev-requirements for Matomo. We did encounter an issue where partitioning the tables would have helped our usecase.

In some rare cases, our serverless DB ran out of locale storage: SQLSTATE[HY000]: General error: 3 Error writing file '/rdsdbdata/tmp/MYB7XJVs' (Errcode: 28 - No space left on device). This if of course hard to fix when it is serverless :)

AWS Support gave us two options to solve this:

  • Increase the number of ACUs when expecting queries that consumes a lot of local storage
  • Partition your tables into smaller sizes so they do not consume all the local storage

We opted for option 1 for now, option 2 would be nice to have upstream, and we might implement it but we would rather work on a as-close-to-upstream config as possible.

I'm not sure how option 2 this would impact write or read performance, since it probably depends on the storage backend and database engine. If the DB could read from multiple partitions when doing large queries I would imagine that would help the overall performance, given the storage backend is supportive of it.

@tsteur
Copy link
Member

tsteur commented Jan 21, 2019

You could try to partition based on log_visit.visit_last_action_time and log_link_visit_action.server_time for example but we haven't run any tests yet. Maybe you could ask in the forum or check if other people already talked about it: https://forum.matomo.org

@tsteur
Copy link
Member

tsteur commented Mar 8, 2023

Good post around this: http://mysql.rjweb.org/doc.php/partitionmaint

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo. Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change.
Projects
None yet
Development

No branches or pull requests

3 participants