This addresses the various dead lock issues when using transactions and multiple recorders. #12733
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Release Piwik 3.2.1
Matomo 3.3.0-b2
…o-org#12478)" (matomo-org#12480) This reverts commit 19a7654.
Matomo 3.4.0
Found `piwik`
kachenjr
changed the title
|
Haven't tested it but makes sense 👍 I presume this may be needed for quite a few other parts as well and may explain why we have seen some queries on options table that take like 50 seconds or so. |
…nd bulk load. Issue matomo-org#6398
|
Thanks for the PR @kachenjr this is a great find! 🎉 I tried to clean the PR a bit (I thought without changing the logic) in da90420 |
|
Actually the test failures are due to another PR, so merging 👍 |
|
Great! Glad I could help! |
InfinityVoid
pushed a commit
to InfinityVoid/matomo
that referenced
this pull request
Oct 11, 2018
…nd multiple recorders. (matomo-org#12733) * visitorGeolocator: output actual changes in debug mode (matomo-org#12478) closes matomo-org#12477 * Revert "visitorGeolocator: output actual changes in debug mode (matomo-org#12478)" (matomo-org#12480) This reverts commit 19a7654. * Fix SystemIntegration test (matomo-org#12726) Found `piwik` * This addresses the various dead lock issues when using transactions and bulk load. Issue matomo-org#6398 * Fix archive test to work with multi recorder fix. * Minor changes and renames
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This addresses the bulk load table dead lock issues like those discussed in issue #6398.
I found that the issue is in logic that marks the date/site as invalid. It does this by inserting a row into the option table for option_name=report_to_invalidate_[idsite]_[date] and value = 1 with option_name being the primary key of the table. With bulk load you end up with longer running transactions than you get normally. Since these transactions are longer running you run the chance of more than one process trying to set this row value and then doing more work before committing the transaction. While that is going on, other processes trying to set this same row have to wait for the lock on the primary key. That's why setting transaction = 0 "fixes" it, and lower recorder numbers make it less likely to happen.
The solution I took for this is fairly simple, I've tagged the pid on the end of the option_name value. This means each process ( since php doesn't have threads ) will write a unique row, and thus not have to wait on any other processes to commit. The rest of the invalidation logic basically works as is, other than changing a couple of the queries to use a Like syntax instead of an exact match. We're able to get multiple recorders with lots of records working very reliably. I've only tested this on Linux ( AWS linux ), not sure if the get pid logic is different on other OSes.
Let me know if you have questions or issues that I've missed with this.
Thanks!