Do not perform an inner join when looking for archives to purge, it does not scale. #8331
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…his can make Piwik completely unresponsive and can result in invalidated archives never being purged.
|
Very nice! I learnt something new about the Power of GROUP_CONCAT. Note: this code is covered by System test: ArchivePurgerTest |
mattab
pushed a commit
that referenced
this pull request
Jul 12, 2015
Do not perform an inner join when looking for archives to purge, it does not scale.
|
Great find @diosmosis 👍 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
One user's install appears to become bloated due the INNER JOIN in Piwik\DataAccess\Model::getInvalidatedArchiveIdsSafeToDelete(). The user has more than one hundred thousand invalid archives. On this install, the query never finished (I let it run for an hour before stopping), and for some reason, the scheduled task that purges invalidated archives returns immediately with an empty result set.
The query I've replaced it w/ takes ~.7s to run when there are ~135,000 invalid archives in a single table.
It's possible this may fix #7181, after fixing this in the user's Piwik, the scheduled task started working. Currently waiting for verification.