refs #8395

I believe we have a few more similar bug reports eg with row evolution #6287

It is likely there would be a regression, eg segments can be urlencoded or just plain etc and there are likely many other cases to consider and to not break any other features / reports that handle the data differently. Fixing this one similar to the other URL related issues will be tricky.

Yeah, I realize that, so I opened a discussion first. I will try to introduce some PR soon. Maybe we can work something out. Thx for looking into this.

I wanted to let you know, that I experienced similar problems. Some of our tracked URLs contain actual percent signs (because in some cases we encode special chars beforehand), see this example of an URL containing an emoji:

http://our.app/path-%f0%9f%98%83

Tracking works correctly, however, the record gets stored in the database actually containing the encoded form of the emoji (which is expected as I sent it this way to the tracking API. I could also sent the record in UTF-8 format without previous HTML encoding, then it will get stored in its UTF-8 representation - given the database supports for example utf8mb4 - again also as expected).

At first, everything looks fine when I open the reports in the Piwik frontend. However, in the displayed URLs the encoded emoji actually gets displayed in the graphical form '😃' which in my opinion is the first sign that something might be wrong. As the URL was sent in encoded form, it should also be displayed as that in the reports.

Then, when I want to see the segmented visitor log or try to get reporting data using the reporting API I finally run into real trouble. In the request sent from the piwik frontend, the requested segment in the query string looks like this:

?segment=pageUrl==http%3A%2F%2Four.app%2Fpath-%F0%9F%98%83

That is actually wrong, I would expect the emoji to be double encoded at this point like that:

?segment=pageUrl==http%3A%2F%2Four.app%2Fpath-%25F0%259F%2598%2583

Even more intriguing is that when I try to request the information using the API I actually do double encode the emoji but still don't get the data I want.
I inspected the actual SQL queries performed during the request and found out that it makes no difference whether I double encode or not, the query always looks for the UTF-8 representation. This is part of the SQL query performed in both cases:

SELECT idaction FROM piwik_log_action WHERE ( name LIKE CONCAT('our.app/path-😃', '%') AND type = 1 )'

Seemingly, in the PHP backend, piwik does multiple decoding rounds, resulting in the emoji encoding being transformed from %25F0%259F%2598%2583 to %F0%9F%98%83 to 😃. That's what I assumed anyway.

Going from there I then tried successively adding more and more encoding rounds to my request and finally got to a state that worked when I actually performed 5 (!!) encoding steps, ending up with this:

?segment=pageUrl==http%3A%2F%2Four.app%2Fpath-%25252525f0%252525259f%2525252598%2525252583.

This finally returns sensible data. To me it seems there is something really broken here with Piwik's assumptions about the encoding states.

Thanks for contributing to this issue. As it has been a few months since the last activity and we believe this is likely not an issue anymore, we will now close this. If that's not the case, please do feel free to either reopen this issue or open a new one. We will gladly take a look again!