Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve memory management in DataTable/CSV #19158

Closed
JasonMortonNZ opened this issue Apr 29, 2022 · 1 comment · Fixed by #19315
Closed

Improve memory management in DataTable/CSV #19158

JasonMortonNZ opened this issue Apr 29, 2022 · 1 comment · Fixed by #19315
Assignees
@sgiehl
Labels
Stability For issues that make Matomo more stable and reliable to run for sys admins.
Milestone
4.11.0

Comments

@JasonMortonNZ
Copy link
Contributor

We are seeing this error occur more regularly now, particularly with CSV format. It tends to appear more when there are visits with many actions, forms, media, etc.

{
    "type": 1,
    "message": "Out of memory (allocated 1982181376) (tried to allocate 163840 bytes)",
    "file": "/var/www/html/core/DataTable/Renderer/Csv.php",
    "line": 199,
    "request_id": "4ad9e",
    "backtrace": " on /var/www/html/core/DataTable/Renderer/Csv.php(199)\n",
    "safemode_backtrace": "#0 [internal function]: Piwik\\Plugins\\Cloud\\Controller->safemode(Array)\n#1 /core/FrontController.php(626): call_user_func_array(Array, Array)\n#2 /core/FrontController.php(168): Piwik\\FrontController->doDispatch('Cloud', 'safemode', Array)\n#3 /core/FrontController.php(99): Piwik\\FrontController->dispatch('CorePluginsAdmi...', 'safemode', Array)\n#4 /core/FrontController.php(274): Piwik\\FrontController::(Array)\n#5 [internal function]: Piwik\\FrontController::triggerSafeModeWhenError()\n#6 {main}"
}

Also listing the API calls here in case these are helpful :)

GET: {
    "module": "API",
    "format": "TSV",
    "idSite": "229",
    "period": "range",
    "date": "2022-04-11,2022-04-15",
    "method": "Live.getLastVisitsDetails",
    "filter_limit": "10000",
    "expanded": "1",
    "translateColumnNames": "1",
    "language": "en",
    "token_auth": "XYZANONYMIZED",
    "force_api_session": "1",
    "format_metrics": "bc"
}

POST: {
    "token_auth": "XYZANONYMIZED",
    "module": "API",
    "format": "TSV",
    "idSite": "229",
    "period": "range",
    "date": "2022-04-11,2022-04-15",
    "method": "Live.getLastVisitsDetails",
    "filter_limit": "10000",
    "expanded": "1",
    "translateColumnNames": "1",
    "language": "en",
    "force_api_session": "1",
    "format_metrics": "bc"
}
@JasonMortonNZ JasonMortonNZ added the Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. label Apr 29, 2022
@sgiehl
Copy link
Member

sgiehl commented Apr 29, 2022

Hi @JasonMortonNZ
Are you able to reproduce that? Could you maybe check if this changes solve that maybe:
https://github.com/matomo-org/matomo/compare/improvecsvexport?expand=1

@sgiehl sgiehl added Stability For issues that make Matomo more stable and reliable to run for sys admins. and removed Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. labels Jun 3, 2022
@sgiehl sgiehl self-assigned this Jun 3, 2022
@sgiehl sgiehl added this to the 4.11.0 milestone Jun 3, 2022
@sgiehl sgiehl mentioned this issue Jun 3, 2022
Merged
11 tasks
@justinvelluppillai justinvelluppillai changed the title Out of memory - DataTable/CSV Improve memory management in DataTable/CSV Jul 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sgiehl sgiehl

Labels
Stability For issues that make Matomo more stable and reliable to run for sys admins.
Projects
None yet
Milestone
4.11.0
Development

Successfully merging a pull request may close this issue.

Improves memory usage of CSV exporter matomo-org/matomo
2 participants
@sgiehl @JasonMortonNZ