Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

core:purge-old-archive-data OOM's if table is too large #14836

Closed
paladox opened this issue Aug 31, 2019 · 3 comments
Closed

core:purge-old-archive-data OOM's if table is too large #14836

paladox opened this issue Aug 31, 2019 · 3 comments
Labels
answered For when a question was asked and we referred to forum or answered it.
Milestone
3.12.0

Comments

@paladox
Copy link
Contributor

paladox commented Aug 31, 2019

Hi, i'm aware that if certain requests in core:archive fail, it builds up over time in the archive table. I'm trying to run core:purge-old-archive-data to see if it'll make the table smaller, but it OOMs because the table is too large. Could the script be improved to not OOM? Or could the core:archive command be improved not to fill the archive table like it does if a request fails?

For reference here's the sizes for the tables (between july and august)

-rw-rw---- 1 mysql mysql 2.4K Aug 26 18:39 piwik_archive_blob_2019_07.frm
-rw-rw---- 1 mysql mysql 652M Aug 26 18:40 piwik_archive_blob_2019_07.ibd
-rw-rw---- 1 mysql mysql 2.4K Aug 26 22:16 piwik_archive_blob_2019_08.frm
-rw-rw---- 1 mysql mysql 3.8G Aug 31 12:52 piwik_archive_blob_2019_08.ibd

-rw-rw---- 1 mysql mysql 2.8K Aug 26 18:45 piwik_archive_numeric_2019_07.frm
-rw-rw---- 1 mysql mysql 288M Aug 26 18:45 piwik_archive_numeric_2019_07.ibd
-rw-rw---- 1 mysql mysql 2.8K Aug 26 22:15 piwik_archive_numeric_2019_08.frm
-rw-rw---- 1 mysql mysql 1.4G Aug 31 12:52 piwik_archive_numeric_2019_08.ibd
@tsteur
Copy link
Member

tsteur commented Sep 1, 2019

What's your memory set to?

@paladox
Copy link
Contributor Author

paladox commented Sep 1, 2019

We have 2gb of ram with most of it being used by redis. The memory we have set is 128M.

@tsteur
Copy link
Member

tsteur commented Sep 2, 2019

@paladox I recommend you set it to 256MB then. We would maybe start investigating if it was more than 512MB or so. Anything less than 256MB doesn't worry us all that much. Especially considering this is not executed very often etc. If you're keen on providing a PR that has same functionality but uses less memory we'd be happy to give the PR a review 👍

@tsteur tsteur closed this as completed Sep 2, 2019
@tsteur tsteur added the answered For when a question was asked and we referred to forum or answered it. label Sep 2, 2019
@mattab mattab added this to the 3.12.0 milestone Oct 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
answered For when a question was asked and we referred to forum or answered it.
Projects
None yet
Milestone
3.12.0
Development

No branches or pull requests
3 participants
@tsteur @mattab @paladox