Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

assign more accurate content-type to CSV output from Reporting API #8898

Closed
mirrorstage opened this issue Oct 2, 2015 · 4 comments
Closed
Labels
Bug For errors / faults / flaws / inconsistencies etc. worksforme The issue cannot be reproduced and things work as intended.

Comments

@mirrorstage
Copy link

It seems like the CSVs produced by the Reporting API are not "true" CSV: they come as Content-Type: application/vnd.ms-excel/ This is fine for those who want to work in Excel, but other use cases the files cause issues. For example, if you try to use one of these in R, attempts to read it into memory as an object fail.

One suggestion: allow text/csv as the content-type.

I've written up my issues found with Piwik CSVs and R here, with more details about what appears to make R choke: http://forum.piwik.org/read.php?2,129529 Happy to append them to this issue report if needed.

@MCMic
Copy link

MCMic commented Oct 15, 2015

I got kind of the same problem.
The weird extra bytes at the beginning of CSV answer from the API are causing a lot of tools to crash or say the file is broken.
Also, the encoding is not UTF-8 and I need to convert it. And I have a problem with line endings too, which might be linked.

I end up writing a PHP script getting the PHP serialized export and creating the CSV myself :-/

@mirrorstage
Copy link
Author

We found a fix after changing two defaults in the file Csv.php in the Piwik package.
Filepath: ~/piwik/piwik/plugins/API/Renderer/Csv.php

In line 34 $convertToUnicode = Common::getRequestVar('convertToUnicode', true, 'int', $this->request);
change true to false and

in Line 57 Common::sendHeader("Content-Type: application/vnd.ms-excel", true);
change application/vnd.ms-excel to text/csv

We changed the content-type first, but that didn't help with the extra bytes (a UTF-16 BOM, I think). Changing the convertToUnicode flag worked.

However, we didn't reset the content-type to the default before we changed the flag. That means we didn't exactly isolate the problem, so I can't say whether this is solely a unicode issue or whether the unicode flag and the content-type default are interacting somehow. If we have some time in the next few days, I'll see if we can test it.

@mattab
Copy link
Member

mattab commented Nov 26, 2015

Thanks for the suggestion. Maybe we could improve this in Piwik 3.0.0! I think it's very important that CSV export create CSV files that work well for all tools. But we need to make sure our CSV files can also contain unicode exotic characters (which is the reason we have implemented this 'convert to unicode feature')

@mattab mattab added this to the 3.0.0 milestone Nov 26, 2015
@mattab mattab added the Bug For errors / faults / flaws / inconsistencies etc. label Nov 26, 2015
@mattab
Copy link
Member

mattab commented Dec 10, 2023

Thank you for the report 👍 To likely fix the issue, could you upgrade to the latest Matomo version?
see https://matomo.org/docs/update/

if you still experience this issue afterwards, please re-open this and post a comment. Thanks!

@mattab mattab closed this as not planned Won't fix, can't repro, duplicate, stale Dec 10, 2023
@sgiehl sgiehl added the worksforme The issue cannot be reproduced and things work as intended. label Dec 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc. worksforme The issue cannot be reproduced and things work as intended.
Projects
None yet
Development

No branches or pull requests

4 participants