Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Field width 3356450096 is too long in core/DataTable.php #18544

Closed
tsteur opened this issue Dec 26, 2021 · 7 comments · Fixed by #21047
Closed

Field width 3356450096 is too long in core/DataTable.php #18544

tsteur opened this issue Dec 26, 2021 · 7 comments · Fixed by #21047
Assignees
Labels
Bug For errors / faults / flaws / inconsistencies etc. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Milestone

Comments

@tsteur
Copy link
Member

tsteur commented Dec 26, 2021

Error: {"type":1,"message":"Field width 3356450096 is too long","file":"/var/www/html/core/DataTable.php","line":1586,"request_id":"2b6ce","backtrace":" on /var/www/html/core/DataTable.php(1586)\n","safemode_backtrace":"#0 [internal function]: Piwik\Plugins\MyPlugin\Controller->safemode(Array)\n#1 /core/FrontController.php(619): call_user_func_array(Array, Array)\n#2 /core/FrontController.php(168): Piwik\FrontController->doDispatch('MyPlugin', 'safemode', Array)\n#3 /core/FrontController.php(99): Piwik\FrontController->dispatch('CorePluginsAdmi...', 'safemode', Array)\n#4 /core/FrontController.php(274): Piwik\FrontController::(Array)\n#5 [internal function]: Piwik\FrontController::triggerSafeModeWhenError()\n#6 {main}"}

URL: https://example.matomo.org/index.php?date=previous24&period=month&expanded=1&filter_limit=-1&force_api_session=1&format=CSV&idSite=4&language=fr&method=API.getProcessedReport&apiModule=Referrers&apiAction=getReferrerType&module=API&period=month&segment=&translateColumnNames=1&convertToUnicode=0&token_auth=....

No referrer

GET: {"date":"previous24","period":"month","expanded":"1","filter_limit":"-1","force_api_session":"1","format":"CSV","idSite":"4","language":"fr","method":"API.getProcessedReport","apiModule":"Referrers","apiAction":"getReferrerType","module":"API","segment":"","translateColumnNames":"1","convertToUnicode":"0","token_auth":"....","format_metrics":"bc"}

POST: {"token_auth":"....","date":"previous24","period":"month","expanded":"1","filter_limit":"-1","force_api_session":"1","format":"CSV","idSite":"4","language":"fr","method":"API.getProcessedReport","apiModule":"Referrers","apiAction":"getReferrerType","module":"API","segment":"","translateColumnNames":"1","convertToUnicode":"0","format_metrics":"bc"}

@tsteur tsteur added the Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. label Dec 26, 2021
@sgiehl
Copy link
Member

sgiehl commented Nov 9, 2022

@tsteur I think that error actually only occurs as API.getProcessedReport can't be exported as a CSV. Normally I guess this part should throw an exception as the data structure can't be converted:

throw new Exception(sprintf($exceptionText, var_export($array, true)));

But it tries to add the full array content to the exception message. Locally that ends up for me in a memory exhausted error. Maybe the message above would occur if I would have a more memory available.

We maybe should change the error messages in that case, so it no longer tries to add the full data content.

@sgiehl sgiehl added Bug For errors / faults / flaws / inconsistencies etc. and removed Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. labels Nov 9, 2022
@sgiehl sgiehl added this to the For Prioritization milestone Nov 9, 2022
@tsteur
Copy link
Member Author

tsteur commented Nov 9, 2022

Truncating the exported array makes sense and could be a good quick fix 👍 We could limit it to the first 1M characters or so

@sgiehl
Copy link
Member

sgiehl commented Nov 10, 2022

Actually I'm not sure if it's worth to include it at all. The variable is an array, so doing a var_export on it will double the amount of used memory for it. So we would need to truncate the array before dumping it, but hard to say how big the dump will be without looking through the array content and its structure.

@tsteur
Copy link
Member Author

tsteur commented Nov 10, 2022

Not sure if I understand correctly. I thought in this case it's not a memory issue but maybe a limitation in sprintf?
If there's a memory issue or so, we could otherwise always extract the first say 1K entries and then do the var_export?

@sgiehl
Copy link
Member

sgiehl commented Nov 10, 2022

I think the problem is, that the array is too big in some cases. But we can't know which part of the array is how big. So using the first 1000 entries in the array might still be too big, depending on the complexity of the array.

@tsteur
Copy link
Member Author

tsteur commented Jul 20, 2023

We had the same problem again

Error: {"type":1,"message":"Field width 2699393714 is too long","file":"/core/DataTable.php","line":1587,

Request:

GET: {"expanded":"1","filter_limit":"100","format":"CSV","flat":"1","date":"2022-01-01,yesterday","format_metrics":"1","idGoal":"","idSite":"29","language":"fr","method":"API.getProcessedReport","apiModule":"Referrers","apiAction":"getReferrerType","module":"API","period":"day","segment":"","showAllGoalSpecificMetrics":"0","token_auth":"XYZANONYMIZED","translateColumnNames":"1","convertToUnicode":"0"}

POST: {"token_auth":"XYZANONYMIZED","expanded":"1","filter_limit":"100","format":"CSV","flat":"1","date":"2022-01-01,yesterday","format_metrics":"1","idGoal":"","idSite":"29","language":"fr","method":"API.getProcessedReport","apiModule":"Referrers","apiAction":"getReferrerType","module":"API","period":"day","segment":"","showAllGoalSpecificMetrics":"0","translateColumnNames":"1","convertToUnicode":"0"}

@sgiehl
Copy link
Member

sgiehl commented Jul 21, 2023

I'll prepare a PR to adjust the exception messages. I guess there is no much value in printing the whole content in the exception anyway.

@sgiehl sgiehl self-assigned this Jul 21, 2023
@sgiehl sgiehl modified the milestones: For Prioritization, 5.0.0 Jul 21, 2023
@sgiehl sgiehl added the not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. label Aug 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants