Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

daily report does not support unicode characters #18465

Closed
mattmary opened this issue Dec 7, 2021 · 10 comments · Fixed by #18577
Closed

daily report does not support unicode characters #18465

mattmary opened this issue Dec 7, 2021 · 10 comments · Fixed by #18577
Assignees
Labels
not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. Regression Indicates a feature used to work in a certain way but it no longer does even though it should.
Milestone

Comments

@mattmary
Copy link
Contributor

mattmary commented Dec 7, 2021

when running matomo on websites which have special characters such as japanese or chinese characters, we are not able to get them in the daily report even if we follow https://matomo.org/faq/how-to-install/faq_142/.

Expected Behavior

Having the characters displayed in the PDF report

Current Behavior

Current unicode characters are not displayed in the PDF report

Steps to Reproduce (for Bugs)

Create a webpage with a title in chinese or in japanese.
Check in the report that these characters are well displayed in the page title report.
create a daily report with page titles.
check the generated pdf that the characters for the chinese page are not well displayed.

@mattmary mattmary added the Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. label Dec 7, 2021
@froger-me
Copy link

froger-me commented Dec 8, 2021

In app/core/ReportRenderer.php line 24:

const DEFAULT_REPORT_FONT_FAMILY = 'dejavusans';
  • The font family dejavusans doesn't support CJK glyphs
  • The font family is shipped with tcpdf ; no font family shipped with tcpdf in app/vendor/tecnickcom/tcpdf/fonts support CJK glyphs unless it's set according to Matomo chosen locale (no support for mixed content)
  • changing dejavusans to droidsansfallback in app/core/ReportRenderer.php line 24 and adding the corresponding font files to app/vendor/tecnickcom/tcpdf/fonts result in reports showing CJK glyphs properly

Because no TTF or OTF font can accommodate all 100k+ glyphs (limit 64k), a suggestion would be to allow font selection/upload for different language combinations in settings to override the automatically chosen font per language.

@tsteur tsteur added the Regression Indicates a feature used to work in a certain way but it no longer does even though it should. label Dec 8, 2021
@tsteur
Copy link
Member

tsteur commented Dec 8, 2021

I'll mark this for now as a regression as I think this used to work?

@tsteur tsteur added this to the 4.7.0 milestone Dec 8, 2021
@froger-me
Copy link

froger-me commented Dec 8, 2021

Note in case there are discrepancies between different shipped versions: I am using Matomo Analytics - Ethical Stats. Powerful Insights. v4.5.0 for WordPress
The website's language is English, but with mixed language content.

@peterhashair peterhashair self-assigned this Jan 5, 2022
@peterhashair
Copy link
Contributor

peterhashair commented Jan 5, 2022

I guess, to view the report with the correct format need to change the Matomo language to the same language that the report has. how to change language is here. https://matomo.org/faq/how-to/how-can-i-change-the-language-of-matomo-as-a-user/
it actually take the last param in the URL as laguage in my case is language=en
/index.php?module=API&action=index&idSite=1&period=day&date=2022-01-05&method=ScheduledReports.generateReport&idReport=1&outputType=3&language=en

It seems like this is not a regression. More like a FAQ. Or we can add a config variable in the config.ini.php called report_lang @tsteur any suggestion?

@froger-me
Copy link

What about when a user has mixed language content? The base language is English, but the content is bilingual, primarily English with some Chinese characters, maybe, for example? The interface language would need to be English for the user (so language=en seems right), but the Chinese characters supported nonetheless to handle displaying the content, like the behavior in a browser?

@peterhashair
Copy link
Contributor

@froger-me, right, good question. I guess we should import the https://matomo.org/faq/how-to-install/faq_142/ fonts here to tcpdf Lib, that should fix it. I will give a try, see if that works.

@peterhashair
Copy link
Contributor

peterhashair commented Jan 11, 2022

@@froger-me Update this issue, it should be in the next release very soon, it use the unifont.ttf, https://matomo.org/faq/how-to-install/faq_142/ If you like a different font, I guess just need to overwrite unifont.ttf

@froger-me
Copy link

Great, thanks for the update!

As a suggestion for possible further improvement - how difficult would it be to set the font in config.ini.php so that it doesn't break at update? I specifically have the WordPress plugin in mind, which I think replaces all the files entirely during plugin update (but not the config.ini.php because it's located outside of the plugin directory) - but that may be a valid concern for matomo as a whole.

It seems only 3 files would need what it looks like minor changes in a few places - calling the Piwik\Config\Config::getInstance(), test for the existence of the config in the file, and if not, use a default one:

app/core/Mail/EmailStyles.php
app/core/ReportRenderer/Pdf.php
app/core/ReportRenderer.php

I might be totally off base here, so please ignore if it's irrelevant.

@peterhashair
Copy link
Contributor

@froger-me that make sense to me, set up fonts in the config will be easier, but I can't make the judgment call. Can I copy the request to another tickets category as an enhancement (new feature), which I am sure the management team will pick up,

@froger-me
Copy link

Can I copy the request to another tickets category as an enhancement (new feature), which I am sure the management team will pick up

Absolutely :) !

@justinvelluppillai justinvelluppillai added the not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. label Feb 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. Regression Indicates a feature used to work in a certain way but it no longer does even though it should.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants