New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Lots of piwik_option inserts while archiving when there are thousands of sites #9753
Comments
It's a matter of \Piwik\CronArchive\SharedSiteIds::getNextSiteId(). |
If it's caused by If there are like 10.000 sites, shouldn't there be like max 10.000 queries to update the option table entry? To put this in relation we might execute much more inserts for archive record entries (eg like 100 per site which would be 1mio). Is it actually inserting or updating? Probably doesn't make a difference :) |
Standard insert or: |
The workaround could be to force-idsites one by one, but this makes it really complicated if you want to start a few threads.. |
👍 |
Summary of what we discussed: We already have various caches like
We are aware that several servers could request the same idSite at the same time because we do not have a proper lock in place but this is ok for now. We need to ideally fix the problem that multiple archives at the same time for same idSite could be created and slow down each other. |
Background:
Issue:
The size of binlog files used to replicate data is so big, that the slave DB is not able to process them. Currently, after 2 minutes we have 400 huge queries to
piwik_option
to update this value. Situation will be even worse when running a few archiving threads which is a must have when you have thousands of sites.Idea:
Maybe we could use Redis to store this option and maybe even all Piwik settings?
The text was updated successfully, but these errors were encountered: