Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make log_action deletion require no table locking #13872

Open
diosmosis opened this issue Dec 18, 2018 · 6 comments
Open

Make log_action deletion require no table locking #13872

diosmosis opened this issue Dec 18, 2018 · 6 comments
Labels
c: Performance For when we could improve the performance / speed of Matomo. c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. Stability For issues that make Matomo more stable and reliable to run for sys admins.

Comments

@diosmosis
Copy link
Member

Currently when purging dangling references from the log_action table, we lock it in order to make sure we don't accidentally delete a row that ends up being referenced during tracking. This is problematic, since purging log_action can take a long time, and blocks tracking while it is running.

The proposed solution is to remove the need for locking by:

  • adding a new action type, ACTION_DELETED. the tracker cannot use an action whose type is ACTION_DELETED
  • when purging, running an update to set the type of all dangling actions to ACTION_DELETED. then running queries to delete all ACTION_DELETED actions.

Issues w/ concurrency must be reviewed carefully when implementing. Eg, we must ensure situations like the following do not occur:

  • update query starts, one action is found to be dangling
  • tracker gets a request for that action, decides to use it
  • update query sets type to ACTION_DELETED
  • tracker uses the idaction, which gets deleted resulting in a broken log

or:

  • tracker gets a request for an action that is currently dangling, and decides to use it
  • update query sees that it is dangling before a log is inserted and sets type to ACTION_DELETED
  • tracker inserts log using idaction which gets deleted resulting in a broken log
@diosmosis diosmosis added the c: Performance For when we could improve the performance / speed of Matomo. label Dec 18, 2018
@diosmosis diosmosis added this to the 3.9.0 milestone Dec 18, 2018
@diosmosis diosmosis added the duplicate For issues that already existed in our issue tracker and were reported previously. label Dec 18, 2018
@diosmosis diosmosis removed this from the 3.9.0 milestone Dec 18, 2018
@EreMaijala
Copy link
Contributor

Which issue is this a duplicate of?

@tsteur
Copy link
Member

tsteur commented Sep 29, 2019

@EreMaijala unfortunately I can't find the issue right now.

@diosmosis do you maybe remember?

@diosmosis
Copy link
Member Author

@tsteur No, it was mentioned in slack but we don't have a paid account so we can't look that far back.

@EreMaijala
Copy link
Contributor

@diosmosis, @tsteur Well, may I suggest reopening this one until the actual duplicate issue is found?

@tsteur
Copy link
Member

tsteur commented Oct 2, 2019

Sure 👍

@tsteur tsteur reopened this Oct 2, 2019
@tsteur tsteur removed the duplicate For issues that already existed in our issue tracker and were reported previously. label Oct 2, 2019
@mattab mattab added this to the 4.3.0 milestone Jul 23, 2020
@mattab
Copy link
Member

mattab commented Jul 23, 2020

Regularly people report issues with this locking mechanism. It would make Matomo that bit more stable if log_action deletion required no table locking. The workaround currently is to run the process less often for example once a year with delete_logs_unused_actions_schedule_lowest_interval = 365 or once a month. But that only postpones the problem or make it happen less often. (Also deleting unused actions less often means that the table has more data, and it takes longer to insert into it, update indexes, makes tracking a little slower)

@mattab mattab added the Stability For issues that make Matomo more stable and reliable to run for sys admins. label Dec 11, 2023
@mattab mattab added the c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. label Dec 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Performance For when we could improve the performance / speed of Matomo. c: Platform For Matomo platform changes that aren't impacting any of our APIs but improve the core itself. Stability For issues that make Matomo more stable and reliable to run for sys admins.
Projects
None yet
Development

No branches or pull requests

4 participants