The string to escape is not a valid UTF-8 string in "@CoreHome/getDefaultIndexView.twig". #4410
Comments
|
Since there is a work-around available, I decrease priority of this ticket. If you experience this bug, please comment here! we would like to hear from more users having the issue. |
|
Increasing priority as user sent us the database by email, so let's try to replicate! Please report in this issue if you experience this bug, we need more report to undertand it. |
|
I had the same problem after our hoster Mittwald provided an update from Piwik 2.10 to 2.13.1. I think I could track it down. The problem are umlaute in goal name. I renamed the goal with an umlaut in the DB and afterwards Piwik frontend was working again for this website, too. |
|
I hit this problem on a fresh installation. Piwik reports:
I checked the template files for the plugin, and every thing looks fine. The error reported suggests that the problem may exist in the |
|
Hi! After upgrading from 2.15.0 to 2.16.0 viewing a segment report comes up with: [code] There are several segment reports that use umlauts. They worked just the same in 2.15.0 (!!!), and in the normal backend they show up without a problem in dropdowns. What's happening? Googled a bit, found to insert 'charset = "utf8"' to config.ini.php but that doesn't change anything. I don't know Twig, how can I debug exactly what UTF-8 string is making trouble here?! |
|
Absolutely same issue after updating to 2.16.0. Edit: every segmented report fails, not only by userId. |
|
To the devs: I'd love to debug, can you give a pointer where in PHP-scope this line 7 gets filled, or to see the actually "bad" content? I don't know where to start looking. |
|
+1, ready to debug this issue, just give some hint |
|
I had the same issu after migrating an existing piwik instance to a another server. We found out that two things resolved this issu:
Sorry, but unfortunately the second option doesn't work for me. |
|
@schwindelbub @VorobeY1326 @garvinhicking @anEffingChamp @hdi-kw Please test this patch: https://github.com/piwik/piwik/pull/9926/files Does it fix the issue for you? @schwindelbub thank you for the tip re: |
|
Sadly no change for me, I also forced default_charset in the php.ini. I'd really love to understand which string exactly twig is trying to escape in getDefaultIndexView.twig at line 7. There must be some way to intercept the actual string so that I can understand where it comes from, and in which charset it is?! |
|
For me there is also no change with this patch. |
|
I find another problem: If a visitor comes from a google search with a "ß" in the search term, this term is broken in the report. For example: Every other Umlaut works fine. |
|
As an hint: I migrated from |
|
I found a workaround that fixes the crash: Just set a comment around the Template.nextToCalendar in the twig template getDefaultIndexView.twig
I have no idea what this line does :-/ But it works without :-) |
|
@tsteur I'm using PHP 5.6.8 on Windows server / IIS. Current system locale is russian. |
|
Having the same issues here with IIS and PHP 5.5.33, german locale |
|
Maybe a random idea as it was mentioned it works with English language. I kind of want to find out whether it's related to the language file or eg the set locale. Can someone replace the English language file with the German language file, switch language to English and see if it works? Something like this mv lang/en.json lang/en.json.backup
cp lang/de.json lang/en.jsonThen switch to English language and reload. You might also need to clear the cache directory in between: rm -rf tmp/cache/*After the test you can restore the correct english file by executing something like mv lang/en.json.backup lang/en.jsonor cp lang/en.json.backup lang/en.json
rm lang/en.json.backup |
|
Nice idea. I'll test, but it'll take some time. Just to be sure: the central lang/de.json file, not a lang file within a plugin directory, yes?
|
|
Yes, the string that contains the previously mentioned |
|
@tsteur As you can see here, the string is not only in the |
True. I had a look in the code though and for this particular widget it should use the string from that file. |
|
Installed fresh 2.16.0 piwik to my local computer (Windows 7, IIS) and the issue easily reproduced. Just logged several actions, added one segment named "IE 11", clicked "save and use it" and voila — crash. Language of piwik is russian. |
|
Can you give #4410 (comment) a try or give us access to a server where we can debug the problem maybe? If so please let us know via email: hello at piwik.org. |
mattab
added
the
Major
|
@tsteur Sorry it took some time. I was able to test now. Copying the english language file over the german one fixes the problem as well. So I guess that the high-byte NON-ASCII char there in "enthält" must cause some sort of trouble in the PHP/IIS environment. Strangely, many other occurences of the german language file cause no problem at all, so I guess this ->getHumanReadable() method maybe double-encodes the UTF-8 input at some point? |
|
Thanks for that @garvinhicking In https://github.com/piwik/piwik/blob/2.16.1/plugins/SegmentEditor/templates/_segmentSelector.twig#L2 can you try to replace |
|
@tsteur This patch has no effect on my PIWIK (now 2.16.1). I still get the error, if the profile language ist "Deutsch". And sorry, but i can't give you access to my server. |
|
@tsteur Sadly, same for me. Doesn't fix the issue, same error. |
|
+1 with a non-fixed piwik including that patch. |
|
I can't see any double encoding somewhere. Also compared hex codes etc and the character It would be really great if someone could offer access to a server to debug this issue. |
|
@tsteur Ok I solved it: In SegmentFormatter.php the getTranslationForcomparison() method uses "strtolower" to transform "Enthält" into "enthält". For that, it utilizes the "strtolower" PHP function, which is by default* not UTF-8 safe. So it messes up the UTF-8 and returns ISO-8859-1. To do that properly, the mb_strtolower function should be used. The easiest, proper patch would thus be to replace this line in plugins/SegmentEditor/SegmentFormatter.php: with: Note that I did hardcode "UTF-8" there, because I believe Piwik internally always works with UTF-8. If not, the proper way would be to use mb_internal_encoding(charset) at some bootstrap code initially. But grepping through piwik's code, it seems whenever mb_* functions are used, you have hardcoded the UTF-8 charset in the function call. Also note that I do see quite a lot instances of "strtolower" inside other files. You might want to check those to see if they could contain non ISO-8859-1 characters, and use mb_* everywhere, so that the same problem will not show up elsewhere. HTH. (*: Unless mbstring.func_overload is set to overwrite strtolower with mb_strtolower in php.ini, which one should not rely on - that should quite likely be the reason why not every Piwik user is affected by this issue when using a language with UTF-8 characters). |
|
@tsteur this solved the problem! Thanks! And it looks - expect the hardcoded UTF-8 - as a good solution. |
|
@garvinhicking Awesome! 👍 Thanks so much! I will issue a PR. Should have seen it while looking at the code but didn't notice. |
|
Great you approve. Afterwards it always gets obvious. Doesn't matter, now we can move forward. Thanks for listening and advising. :-)
|
* Add link to Product changelog in the Developer changelog (#10046) Here is a recent comment from a community member ` another colleague found and pointed me to your changelog for 2.16.1, which was admittedly hard to find, ` which triggered me to add a clear link in our visible changelog root file. * Update FrontController documentation (#10049) * Update link to the overview * Remove deprecated method from example * Add TweakStyle browser icon * Add possibility to execute plugin test suite * 2.16.2-b1 * Fix pagination in datatables when displaying all entries (#10063) * fix #4410 The string to escape is not a valid UTF-8 string (#10050) * Trigger a Goal conversion for "Event matching goals" even when an event is tracked without a URL (#10018) * use idGoal parameter in system tests * Trigger a Goal conversion for "Event matching goals" even when an event is tracked without a URL * Test files for green build * Add new expected test file * do not start a session during tracker mode or if session is not started yet (#10071) Eg could start a session during tracking if a tracker plugin is not compatible with the current Piwik and it would try to trigger a notification because of this. * Implements #9389 - UserId module, that adds a new Users report (#9883) * Implement the UserId module, that adds a new Users report showing all unique user IDs and some aggregated data. Includes reindexing of raw visitors log into aggregated user ids table * Update UserId module: add total_actions, total_events and total_searches columns; format dates * Use an injected model instead of creating a new object * Rework the UserId plugin to use core archiver instead of custom indexer * Users report small changes: - remove logger injection - change archive record name - add low population filter - add all columns visualization - add datatable_archiving_maximum_rows_userid_users configuration, default value is 50000 * Remove unused method parameter * Users report: remove custom visualizations and add data-row-metadata attribute for every row * minor UserId tweaks, enable row evolution for userId etc (#10072) * addding userId plugin description and enable plugin during update (#10073) * Fix processed metrics are not shown in visitor map when viewing cities Eg nb_actions_per_visit bounce_rate avg time per visit Bounce rate seems to always show 100% now but not sure why * some more fixes * fix tests after adding new userid feature * Fix tests * Ignore idvisitor in system tests as it always changes in each test * Skip email tests in case intl extension is not installed * Update ui tests * update submodules * Explain a user should call setSessionCookieTimeout when changing visit length * value should be specified in seconds, not milliseconds * fix example was wrong * language update * allow + in database names during setup (#10091) * only fail when at least 5 domains extensions are failing, so it does not fail every time IANA adds a new domain extension... * Remove travis `group: legacy` and use their latest infrastructure (#10094) * Removed group:legacy from travis.yml template * use latest php 5.3 instead of php 5.3.3 * Run MySQL off the RAM in our CI server to have fast builds * Use travis files from latesst master branch commit * updates travis.yml * language update * refs matomo-org/searchengine-and-social-list#3 improves detection for google searches without keyword * refs matomo-org/searchengine-and-social-list#3 improves detection for yahoo searches without keyword * Fix some typos in comments (found by codespell) (#10120) Signed-off-by: Stefan Weil <sw@weilnetz.de> * refs matomo-org/searchengine-and-social-list#3 improves detection for yahoo japan searches without keyword * fixes #10130 7 days in Evolution over the period when Period is Day * misc: Fix typos (found by codespell) (#10136) Signed-off-by: Stefan Weil <sw@weilnetz.de> * plugins: Fix typos (found by codespell) Signed-off-by: Stefan Weil <sw@weilnetz.de> * submodule update * language update * improve referrer type detection for direct entries ignore fallback detection (same host for referrer and action) if ignore unknown urls is activated for current site * adds some more test cases * update translation command to use ProgressBar instead of deprecated ProgressHelper * Manage Website: Links to website (#10087) * Manage Website: Links to website Add clickable urls for each website as requested in #10066. * Remove extra = and whitespace Updated style * New tab, rel noreferrer, ng-href, commas Changed the urls to open in new tabs, rel=noreferrer, ng-href, and add commas back into the list as requested * Fix pasting error Edit out extra </a> and {{ urls}} * Change div to span Change div to span to have url on the correct line * When only a icon is rendered, the title is not enough to provide an accessibility label, so it an aria-label to the icon so screen reader element lists and such have something to display. * Update urlnormalizer.js (#10171) * Update piwik.js (#10170) * JavaScript SDK: add support to send tracking requests to multiple Piwik servers and/or store in one or multiple websites ID (#10085) * Add support to configure multiple trackers easily * added tests * fix tests * 2.16.2-b2 * Use Common::sendHeader for consistency * Up to date minified piwik.js #10085 * Screenshot UI tests pixel alignment minor change * Escape the value as suggested in the Pr comments. * language update * refs plugin-CustomDimensions#31 fix doc block for custom dimension was wrong * add callback ability to trackPageView (#10172) * Update piwik.js * Update piwik.js * submodule updates * language update * update languages * rebuilt piwik.js * split all tests (#10192) * submodule update * fix some tests * device detector update * Add mention to BrowserStack.com in our readme (#10205) * Check if UserCountry plugin is activated before showing location data * ensure to reset translated categories for each system test (#10210) * link to useful CI build result dashboard and tweaks * in Evolution over the period when Period is Day the limit should be actually 8 instead of 7 See #10130 (comment) * Fix the flags of Switzerland and Nepal Adding a transparent padding ensures the flags will not be stretched to the 16x11 format of all the other flags. * Update the flag of Tokelau * Update the flag of Myanmar * Update the flag of Lesotho * Update the flag of Libya * Update the flag of French Guiana * fixed some typos [ci skip] * fix report order of browser reports * update tests with new order * Update JQuery to the latest 1.12.4 version (#10260) * Update JQuery to the latest 1.* version * UI tests: Updated Jquery to latest version * submodule update * Update jquery to latest 1.*, new release checklist test * language update * Remove uneeded HTML files from a JS library * Put idSite into single quote for support custom string id (#10247) * Put idSite into single quote for support custom id This is small change, but is necessary for #9975 and #4920. On my tests, even with single quote, numeric idSite works fine. * Change unit test expected to support test with strings * Change expected data to support test with strings * submodule update * submodule update * Tweak core:archive output logging to better explain where to find error log depending on server configuration * Adding hyphen for consistency * Composer: remove the disable tls flag, refs #10023 (#10277) * Do not throw an exception when debugging tracker (#10267) This method is only called when Tracker debugging is enabled. It may cause problems if an unknown type is used and then the tracking completely fails which also stops tracking. Debugging info should not really throw exceptions, at least not in this case. * Fix CSV export cannot be used with dates like previousX (#10273) * fixes #9838 - show description bubbles below column if they would be out of viewport if shown above * submodule update * Let plugins provide custom piwik.js tracker tests (#10270) Plugins could provide custom tracker tests by creating a file 'tests/javascript/index.php' within the plugin. Won't be a documented feature for now but could add it to dev changelog if needed. Content within the test file could be eg ``` <script type="text/javascript"> test("MyPiwikTest", function() { expect(1); var tracker = Piwik.getTracker(); equal(typeof tracker.trackGoal, 'function', 'trackGoal' ); }); </script> ``` * Fix segment data disappears from UI after changing auto_archive setting (#10078) * refs #10036 fallback to global done flag if plugin done flag has no archive/data * global done flag might be the same as the plugin done flag if no segment and no range is used * installing touch-punch lib to fix dragging issue on mobiles (#10045) * Reset the list of items in the cart, after tracking an ecommerce conversion (#10279) * Clarify that token_auth must be an admin or super user token. * Reset the list of items in the cart, after tracking an ecommerce conversion But do not reset the list of itms in the cart after tracking an ecommerce cart update. fixes #10252 * Adding changelog note `piwik.js`: after an ecommerce order is tracked using `trackEcommerceOrder`, the items in the cart will now be removed from the JavaScript object. Calling `trackEcommerceCartUpdate` will not remove the items in the cart. * Add possibility to specify a custom row aggregation operation (#10288) * Add possibility to specify a custom row aggregation operation * added test for row metadata aggregations * When JSLint validation failed, explain where to find the error messages refs #10271 (comment) * trackLink does not invoke callback if request is done via GET (#10294) * trackLink does not invoke callback if request is done via GET * Update piwik.js * Make JSlint errors output actually useful to developers * JSLint: better error reporting, correct line numbers #7232 * Improved join generation for segments (#10264) * better join generation for segments * do not update visit if there are no values to be updated * Calling an undefined JS API method should not trigger an error in the console (#10176) * When calling a non existing method, do not fail JS execution but print a useful error message in browser console * Fix * Throw an error so developers can see in the browser console * Up to date minified piwik.js * Improve detection for empty keywords by allowing to define that in search engine definitions * updates piwik/searchengine-and-social-list dependency * refs #9984 - adds confirmation when changing segments definition * refs #9984 - adjust UI test to automatically confirm confirmation * better naming * record actual screen resolution when using JS tracker (#10296) * Add current domain to list of domains automatically only if no path is set (#10051) * Do not add current domain automatically if setDomains is called, might also fix a bug in IE11 * this might fix jslint test * added tests and fixed a bug * fix tests * Up to date piwik.js * Minor changes * Document new internal piwik.js change https://github.com/piwik/piwik/pull/10051/files * Add link to multiple tracker docs in changelog. #10174 * make sure we prefer forwarded proto header over regular header (#10081) * 2.16.2-b3 * Fixes #10117 * Fix typo * Refs #10115 * use better message depending of the availability of archiving triggered by browser * add option to hide segment definition change confirm * Add a test for isInsideAnIFrame method in piwik.js (#10300) #10271 #10105 * Searching recursively in sub-tables should work as expected (#10303) * When a datatable is manipulated in memory, the correct id subtable is stored in metadata and must be fetched from there. * piwik.js: Prepend a slash to all paths such as when users don't specify leading slash for URL path (#10302) * Load the JS file in the test where it is used, to prevent errors running tests on IE8 jslint not working on IE8 it seems * Fix JS error when running tests in edge browser * iframe waits for document ready * Fix the JS tests on Chrome * Fix issue IE * Fix for IE9: wait for window.onload * Fixes #10105 - copied from #10271 by @Kijewski (#10306) * submodule updates * Added API method to get a single goal (#10305) * added method to get a goal * always fetch goal directly to not having to iterate over all goals * added new changelog entry for devs * updates phpunit dependency / submodule updates * submodule updates * Use log tables to decide which tables can be purged (#10304) * use log tables to decide which tables can be purged * make sure to return the number of deleted visits * fix tests * UI tests: Log tables purge * added app manifest (#10321) * Fixes #10276 * Update piwik/network component to 0.2.0 release for IPv6 ranges support #9411 (#10324) * language update * Segment editor, updating segments, logic when to display messages (#10322) * Display the right message when segment is set to real time * Correct logic * UI tests: minor messaging change * Submodules including matomo-org/travis-scripts#28 #10323 * Update integration tests for #10276 * Remove comment * submodule update * submodule updates * Php7 compatibility, Pear archive_tar package #10058 * Update submodules * fixes #10310 - use indexOf polyfill function to avoid errors in older IEs (#10327) * pass instance of both row objects to aggregate callback when aggregating them (#10330) * Update composer libraries, while enforcing the minimum php in composer.json (#10334) * Enforce the Minimum PHP version in composer.json using "platform" feature Added in composer/composer@a57c51e It was found that this flag is very important to use to make sure the package we build and release will be compatible with this PHP version. Otherwise, the minimum PHP version enforced would be the version where our build script runs, which could be PHP 5.5 or PHP 7 which would have caused serious issues. * Update doctrine/cache, monolog,piwik/network, piwik/cache, piwik/php-tracker etc. #10058 * Release checklist to check that composer.json is correctly defined * Use latest travis scripts for matomo-org/travis-scripts@30d7b16 * .travis.yml file is out of date, auto-updating .travis.yml file. * Use latest travis scripts for matomo-org/travis-scripts@63445cf * .travis.yml file is out of date, auto-updating .travis.yml file. * Fetch latest referrer spam blacklist refs #10058 * When Piwik config file becomes un-readable, make it clear that the file still exists but is not readable (#10331) * * In messages, make it more clear whether config file exists and/or is (not) readable * When the config file exists but is not readable, provide users the choice between making the file readable, or starting a fresh install * As long as Piwik is not installed, piwik.php should not return 500 when config file is not readable * Fixes #10283 * UI tests: updated message when config file is not found * Update submodules to latest - refs #10058 (#10335) * 2.16.2-b4 * Refs #10331 #10283 * 2.16.2-b5 * Add require_once to prevent build failure * Add missing file * 2.16.2-b6 * Professional Services for Piwik, neutral branding within Piwik app (#10339) Professional Services for Piwik - neutral branding within Piwik app * 2.16.2-rc1 * Update submodule, and fix to update file * 2.16.2-rc2 * UI tests: fixed typo * language update * add possibility to create ui tests for users with view access only * Added Recommended Drivers During Installation (#10346) * Fixed * Made Recommended Changes * Add support section to plugin json template (#10358) * Add support section to plugin json template As suggested by a developer. The Marketplace has been supporting this for quite a while already. * Added support section to example theme plugin json * added wiki section * added wiki section * Adding couple issues to the changelog #10058 * Do not document until finalised #10270 * Update to Device detector 3.7.1 (#10362) * Device detector 3.7.1 * Fix build * 2.16.2 KABOOM * fix #10361 (#10361) Now only the english string is changed, as translations are handled via https://www.transifex.com/piwik/piwik/ * Add pull request template mentioning PRs should be made against 3.x-dev (#10030) * Do not limit password length (#10349) * Disabled Max Password Length Check * Disabled Max Password Length Check * Reset config files * Translation & other fix * Fixed Test Case for Very Long Passwords * UI tests: #10357 * fixes #10374 generate:visualizationplugin was not working * support section is no longer needed * language update * Update global.ini.php (#10408) * submodule updates * Remove again the Hiring file #9814 * Minor Improvement to description * Fix the integration test and patch Zend Validate * on Travis use PHPUnit from composer * Create index.htm files in all tmp/ sub-folder as an additional safety net (#10414) * Create index.htm files in tmp/ folder as safe net * UI test * silent fail * fix unit test * Minor Improvement to description * Fix release checklist * Fix release checklist * UI test * UI test logic * Actually make methods public to keep BC * Fix fatal error when requesting RSS format (#10407) * Fixes #10399 * add test reproducing issue #10399 * another health check * Fix no required Piwik version was defined (#10413) * Fix no required Piwik version was defined * plugins should specify an upper version constraint to the next major version * PHP 7: when there is a fatal error in API request processing, display the original error in the api response (#10406) * PHP 7: when there is a fatal error in API request processing, display the original error * PHP 7 compatibility * Use Piwik 40.0.0 so that test does not need to be updated soon * UI test * UI tests: plugins not compatible with next version are shown * language update * submodule update * Split piwik.js into multiple files and provide a merged one (#10441) refs #6106 * Fixes #10143 (#10424) * Fix issue when running archiving via the old archive.php path Notice: Use of undefined constant PIWIK_DOCUMENT_ROOT - assumed 'PIWIK_DOCUMENT_ROOT' in plugins/CustomPiwikJs/Commands/UpdateTracker.php * fix tests
Reported in the forum: http://forum.piwik.org/read.php?2,108645
There was proposed solution:
http://forum.piwik.org/read.php?2,108645,page=1#msg-108949
The text was updated successfully, but these errors were encountered: