@Findus23 opened this Issue on June 26th 2020 Member

(I thought I wrote about this before, but couldn't find an issue)

The Chrome team is working on a large change that plans to deprecate the user-agent string.
The current(?) Chrome 83 already freezes the string and Chrome 85 will unify it to one common string for all desktop and one for all mobile users.

As a replacement the client hints standard is proposed: https://github.com/WICG/ua-client-hints

This allows to get similar data about a device already separated by things like brand, browser, os, etc. (meaning device-detector would not be needed anymore).

There is both an HTTP header (that only returns non-vague results if the server requests them) and a JS API for accessing the data.

Things that might complicate this:

the main UA value is not a string, but a list/set

Chrome"; v="73", "ChromiumBasedBrowser"; v="60", "Chromium"; v="73" (and the corresponding dictionary from JS) is a valid response and browser vendors are encouraged to "randomly [include] additional, intentionally incorrect, comma-separated entries with arbitrary ordering".

It seems like at the moment only Chromium-based browsers are working on implementing this standard, but as it seems like these are by far the majority now, that doesn't really matter.

I added this to 4.0 RC for now as it impacts every Matomo user as the most popular user agent will display incorrect results.

https://github.com/mozilla/standards-positions/issues/202#issuecomment-558294095 (Position of the firefox team on this standard)

@tsteur commented on June 28th 2020 Member

BTW @Findus23 just tested using latest Chrome 83 and Chromium 86. Both of them did not send any of the new headers, neither is the new navigator property set to read this through JS, nor did it look like the user agent was frozen. Wonder if these things were delayed eg due to covid 19

@tsteur commented on June 28th 2020 Member
@tsteur commented on July 17th 2020 Member

checked again and seems there's still no update

@tsteur commented on September 3rd 2020 Member

This feature is still in development in Chrome

@sgiehl commented on October 1st 2020 Member

It's at least meanwhile possible to read the values after enabling them in Chrome > 85:
See https://user-agent-client-hints.glitch.me/

@tsteur commented on October 1st 2020 Member

After enabling the experimental flag this worked for me. It was setting by default eg these headers along the tracking request

sec-ch-ua: "\\Not;A\"Brand";v="99", "Google Chrome";v="85", "Chromium";v="85"
sec-ch-ua-mobile: ?0

Platform headers would not be sent. Be great to focus on the other issues first in the RC milestone and towards the end on this one maybe.

Is the plan that we support these new headers in the device detector @sgiehl ? Or maybe we instead make eventually use of the JS api navigator.userAgentData and send this information along the tracking request once we are able to retrieve it so over time we in a few years we might be able to no longer use device detector which be great (as it is taking too much work). Of course depends if other browsers will implement this as well.

And it will be up to Matomo/website owner to send the correct headers so we receive the needed information?

@sgiehl commented on October 2nd 2020 Member

I think I wouldn't add that to device detector actually. We should handle that within Matomo...

@tsteur commented on November 18th 2020 Member

@sgiehl @mattab I'll move this one to Matomo 4.1

@tsteur commented on November 18th 2020 Member

@sgiehl @Findus23 it's not clear to me when they will start freezing user agent. Is it though already worth looking into this what we need to do to make this work?

@sgiehl commented on November 18th 2020 Member

I already had a look on that a few days ago. It's possible to access those client hints using javascript like this:

navigator.userAgentData.getHighEntropyValues(["model", "platform", "platformVersion", "uaFullVersion"])
   .then(function(ua) {
         var model = ua.model;
         var os = ua.platform;
         var osVersion = ua.platformVersion;
         var browserVersion = ua.uaFullVersion;

As you can see it's accessible using a promise only, which will make it a bit more difficult to integrate into piwik.js I think. Additionally I did not yet find any useful information which values the variables can have. I only see the values my Chrome has...

What we need to do on Matomo tracking side is to identify the short codes for browser, os and brand based on the given values. Not sure how easy that will be. But guess we need to see that as soon as some more details on the values are known.

@Findus23 commented on November 18th 2020 Member

I honestly can't find any news on this that is newer than my initial post. It seems like Google has posponed the planned changes. The only news are that Chrome now has an experimental flag for enabling the client hints.

And I think if the proposal is actually implemented as intended by Google¹ then I am not sure if it is useful to record the output in Matomo as it will intentionally give randomized incorrect results. https://wicg.github.io/ua-client-hints/#grease

https://github.com/w3ctag/design-reviews/issues/467 shows a bit more discussion about the topic of freezing ua strings (and the least helpful "graph" I have ever seen).

¹ This proposal is still only created and discussed by Google (and Microsoft Edge) developers, no non-chromium-browser engines have any plans on implementing it.

@Findus23 commented on November 18th 2020 Member

> Will the delay also include freezing the UA string (i.e. will the current UA string be up-to-date until some change in 2021).
That is correct

from https://groups.google.com/a/chromium.org/g/blink-dev/c/-2JIRNMWJ7s/m/yHe4tQNLCgAJ

@tsteur commented on November 18th 2020 Member

👍 let's maybe wait until they announce it again (which I think they will).

@ampaze commented on April 26th 2021

Waiting for the Browsers to start freezing the User Agent before implementing Client Hints is really asking for trouble...

Client Hints are enabled by default in Edge 90 and Chrome 90, it won't be that long before User Agent freezing is here.

@Findus23 commented on April 26th 2021 Member

It would be great if Client Hints could become a browser standard (both https://wicg.github.io/ua-client-hints/ and https://datatracker.ietf.org/doc/html/rfc8942 seem to be a draft and experimental) and be implemented by more than one browser (engine). Also I don't see any specific information that User Agent freezing is no longer delayed indefinitly (https://bugs.chromium.org/p/chromium/issues/detail?id=955620).

But then again it seems like that's not how the Google Monopoly on browser standards works at the moment, so maybe it might be useful to implement this one day before Google suddenly adds it to all browsers like it did with FLoC.

@ampaze commented on April 26th 2021

But then again it seems like that's not how the Google Monopoly on browser standards works at the moment, so maybe it might be useful to implement this one day before Google suddenly adds it to all browsers like it did with FLoC.

That is most likely how it will work out, even if other browsers don't follow. Chrome alone has enough users, that it might be wise to be prepared for a potential UA-Freeze, even if it will be postponed by Google for a while.

@tsteur commented on April 27th 2021 Member

Moved it back to 4.3 milestone 👍

@justinvelluppillai commented on July 16th 2021 Contributor

I've revisited this with @tsteur and it looks like chrome hasn't frozen UA strings yet. We could extract browser version from userAgentData fairly easily. It will require some work to normalise OS and browser information for matomo.

Mozilla has also updated their view on client hints to "harmful" https://github.com/mozilla/standards-positions/issues/552

We will revisit this later.

@tsteur commented on August 22nd 2021 Member

fyi https://blog.chromium.org/2021/05/update-on-user-agent-string-reduction.html says it won't happen this year (2021) for chrome and likely won't come any time soon

@DannyRo22 commented on September 16th 2021

@tsteur This week Chrome published an update for UA Reduction plans with projected timelines and Chrome milestones: https://blog.chromium.org/2021/09/user-agent-reduction-origin-trial-and-dates.html

@tsteur commented on September 16th 2021 Member

Thanks for this @DannyRo22 I'll move the issue in an earlier milestone so we keep monitoring things.

@tsteur commented on October 21st 2021 Member

https://youtu.be/Qq4rRt8BNjM?t=198 talks about this new feature.

There's also now https://developer.chrome.com/blog/user-agent-reduction-origin-trial/

This will enable sites to discover and fix problems before the reduced UA becomes the default behavior in Chrome (the reduction is planned to start in the second quarter of 2022)

@tsteur commented on October 21st 2021 Member

Someone needs to look at these changes, write up a brief of what needs to be done technically to still detect browsers, devices, browser plugins and operating systems. Then we should split this big issue into lots of smaller issues. For example we could have one issue detected to device, another issue detected to operating systems. It's unclear at this stage how we can best split this up.

We might not fully cover this issue yet in 4.7 and we might need to do this over several releases.

This issue might be a lot of work. For example mapping all the browser names to our browser names etc.

@sgiehl commented on January 20th 2022 Member

We are planning to extend the device detector, so it's also able to handle client hints.
So in the future it would take the useragent and an array with the client hints as parameters and return the best result out of the combined results. If there are usable results in the client hints those will be used first, falling back to the useragent results.

So the remaining work on Matomo side would be to somehow gather the client hints so they are available serverside to be passed to the device detector.
Also we might maybe need to consider to include the client hints in the finger print maybe.

I'll report back here once a version of device detector has been release supporting client hints.
In the meantime someone could already start implementing the other required stuff...

@Findus23 commented on January 20th 2022 Member

@sgiehl So does Matomo want to use the JS client hint API and append the data to the matomo.php request? Or ask the browser to send the HTTP headers with the matomo.php request and then use this data server-side?

@sgiehl commented on January 20th 2022 Member

@Findus23 that actually still needs to be defined, but I guess it will be easier to do that in javascript. Guess we couldn't do that with the first tracking request, as we first would need to send the according headers and would need another request to receive the values. Or maybe I misunderstood how that works 🤔

@tsteur commented on January 20th 2022 Member

@sgiehl it be great to define what the other "required stuff" is? What exactly does device detector do and what not?

Also does device detector handle the mapping of the different browser, operating system, ... name to the names we are using?

@sgiehl commented on January 20th 2022 Member

I have just started implementing, but it should take the client hints (either HTTP headers, or the javascript values) and parse them in a way that the results are the same when parsing user agents. So yes, the mapping will be done by device detector.

From what I understood, I would say there are actually only the tasks mentioned above:

  • Change the tracking that it receives the client hints
    Without having a closer look I would say the simplest way would be to adjust the detectBrowserFeatures javascript method, to also fetch the client hints. That way also the fingerprint would include them.
  • Adjust the php code, so it passes the received client hints to device detector (along with the useragent)

Everything else should imho then work out of the box. Or am I missing something?

This Issue was closed on July 4th 2022
Powered by GitHub Issue Mirror