Use navigator.sendBeacon by default and minimise the tracker pause from 500ms to 100ms #6641

Currently Piwik adds a delay on clicks in order to ensure the links are tracked properly. As Chrome has now shipped navigator.sendBeacon can we get it to use this instead of the delay if the browser supports it?

This is nice. Also Firefox implemented it etc

:+1: for giving it a try

Never heard of it until now! This looks very useful :+1:

Tried implementing this into the JS tracker and my browser does fire the events correctly. However it seems the server side is not logging the requests because nothing shows up in the visitor logs.

sendBeacon is POST only and if sending the request as a string it will be transmitted as "text/plain;charset=UTF-8". I saw that piwik POST requests are transmitted as "application/x-www-form-urlencoded; charset=UTF-8". It is possible to use FormData with sendBeacon but it will be transmitted as "multipart/form-data", I did not try this approach. Is there a check or limitation on which content-type the server expects for POST tracking?

Hi there, you can enable tracker debug see:
http://developer.piwik.org/api-reference/tracking-api#debugging-the-tracker

Hi, turned on debug tracking and this is what a sendBeacon post request looks like from Chrome (copied as cURL to show results)

curl 'http://piwik.elysian.se/piwik.php' -H 'Cookie: OVHCDN=R2163973028' -H 'Origin: http://soniccharge.com' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_101) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2239.2 Safari/537.36' -H 'Content-Type: text/plain;charset=UTF-8' -H 'Accept: /_' -H 'Cache-Control: max-age=0' -H 'Referer: http://soniccharge.com/download' -H 'Connection: keep-alive' --data-binary 'action_name=Sonic%20Charge%20-%20download&idsite=2&rec=1&r=289293&h=12&m=53&s=13&url=http%3A%2F%2Fsoniccharge.com%2Fdownload&_id=75819894e2914cd2&_idts=1417780393&_idvc=1&_idn=1&_refts=0&_viewts=1417780393&send_image=0&pdf=1&qt=0&realp=0&wma=0&dir=0&fla=1&java=1&gears=0&ag=0&cookie=1&res=2880x1800&gt_ms=585' --compressed

Response.

<pre>DEBUG Piwik\Common[2014-12-05 11:54:26] [8a63a] Debug enabled - Input parameters:</pre>
<pre>DEBUG Piwik\Common[2014-12-05 11:54:26] [8a63a] array (</pre>
<pre>DEBUG Piwik\Common[2014-12-05 11:54:26] [8a63a] )</pre>
<pre>DEBUG Piwik\Common[2014-12-05 11:54:26] [8a63a] Nothing to notice =&gt; default behaviour</pre>
<pre>DEBUG Piwik\Common[2014-12-05 11:54:26] [8a63a] End of the page.</pre>
<pre>DEBUG Piwik\Common[2014-12-05 11:54:26] [8a63a] array (</pre>
<pre>DEBUG Piwik\Common[2014-12-05 11:54:26] [8a63a]   &<a href='/039'>#039</a>;OVHCDN&<a href='/039'>#039</a>; =&gt; &<a href='/039'>#039</a>;R2163973028&<a href='/039'>#039</a>;,</pre>
<pre>DEBUG Piwik\Common[2014-12-05 11:54:26] [8a63a] )</pre>
<pre>DEBUG Piwik\Common[2014-12-05 11:54:26] [8a63a] Time elapsed: 0.001s</pre>

Looks like the tracker side does not read the posted data only the cookie in the header.

Using sendBeacon with FormData was recognized by the tracker. Now unfortunately it generates a lot of unnecessary data as FormData is transmitted as "multipart/form-data". It also required a small URI to FormData converter but it is still code that we can get rid of if the server side tracker is taught to parse "Content-Type: text/plain;charset=UTF-8".

At least the implementation is now working. =)

Form data example

--data-binary $'------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="action_name"\r\n\r\nSonic Charge - download\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="idsite"\r\n\r\n2\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="rec"\r\n\r\n1\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="r"\r\n\r\n745302\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="h"\r\n\r\n13\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="m"\r\n\r\n43\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="s"\r\n\r\n56\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="url"\r\n\r\nhttp://soniccharge.com/download\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="_id"\r\n\r\n75819894e2914cd2\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="_idts"\r\n\r\n1417780393\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="_idvc"\r\n\r\n2\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="_idn"\r\n\r\n0\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="_refts"\r\n\r\n0\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="_viewts"\r\n\r\n1417781167\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="send_image"\r\n\r\n0\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="pdf"\r\n\r\n1\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="qt"\r\n\r\n0\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="realp"\r\n\r\n0\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="wma"\r\n\r\n0\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="dir"\r\n\r\n0\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="fla"\r\n\r\n1\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="java"\r\n\r\n1\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="gears"\r\n\r\n0\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="ag"\r\n\r\n0\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="cookie"\r\n\r\n1\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="res"\r\n\r\n2880x1800\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T\r\nContent-Disposition: form-data; name="gt_ms"\r\n\r\n691\r\n------WebKitFormBoundaryoWAmoJTVjOZTsg0T--\r\n'

as compared to text/plain version

--data-binary 'action_name=Sonic%20Charge%20-%20download&idsite=2&rec=1&r=289293&h=12&m=53&s=13&url=http%3A%2F%2Fsoniccharge.com%2Fdownload&_id=75819894e2914cd2&_idts=1417780393&_idvc=1&_idn=1&_refts=0&_viewts=1417780393&send_image=0&pdf=1&qt=0&realp=0&wma=0&dir=0&fla=1&java=1&gears=0&ag=0&cookie=1&res=2880x1800&gt_ms=585'

Started looking into adding the text/plain support on the server side implementing php://input just to realize that is exactly what bulk requests already does. Why re-invent the wheel... I'll make a new sendBeacon implementation that uses the bulk-request format instead.

Just FYI: The bulk tracker does currently not check whether there are scheduled tasks to run etc. We might want to enable this at some point in case we merge this (at the latest once sendBeacon is supported by more browsers).

Ah I see. As long as I have visitors without Firefox or Chrome, normal tracking requests till be made as well. I guess this is good enough for now? I have implemented this for all sites I track, mainly because the sendBeacon method (if available) queues the request and returns instantly which avoids round-trip delays on downloads and outlinks.

I do have one concern regarding the server side implementation. When using sendBeacon you do not know exactly when the request is made, it is up to the browser to queue and dispatch these at its earliest convenience. Technically I suspect that multiple requests can reach the server at the same time. I noticed there was a delay in the code to avoid this because multiple visits could be created by mistake. If it is still needed, I am guessing there should be a better solution implemented for this as an arbitrary delay is far from reliable. This should be handled on the server side imho (if it is not done already).

Another thing that comes to my mind: When using Content Tracking we might use the piwik.php tracking request to redirect the user to the URL that was clicked originally. This means we do not track with XHR but instead open piwik.php?idsite....&redirecturl=$clickedUrl which in turn redirects the user to $clickedUrl after the request was tracked. This was a requirement to make "right click => open in new window" etc track the click as well. There is a ticket to implement the possibility to disable this behavior: #6265 . We should make this the default behavior (redirect via piwik.php disabled) to make sendBeacon work by default for everyone. Otherwise those redirects would not work. I think to disable those redirects via piwik.php by default is a good idea anyway.

Good news: sendBeacon is now supported in most major browsers but IE: https://caniuse.com/#search=sendbeacon

We should 100% give this a try and if the feature available in the browser, use it (if tests are good). This would minimize failed requests that don't make it within 500ms while at the same time not delaying the next page load by 500ms.

More information: https://developer.mozilla.org/en-US/docs/Web/API/Navigator/sendBeacon

We could move it into 3.4 or at least 4.0?

Basically, the idea be to set a variable in the unload callback and then for all following (POST and / or GET) tracking requests use sendBeacon instead of the request XmlHttpRequest.

We could likely send also GET requests using sendBeacon as long as the URL parameters are fully in the URL so it can be replayed by log analytics if needed.

A first implementation is done in https://github.com/matomo-org/matomo/pull/12538

However, we should also fully support this and allow users to opt in for always using navigator.sendBeacon if available in the browser. When enabled, we can reduce the configTrackerPause = 500, from 500ms to eg 100ms which makes switching between pages faster when eg a user clicked an outlink etc.

sendBeacon always sends a POST request. I tested it and we could send our regular GET requests via sendBeacon and have all parameters in the URL but this request would be still a POST. We need to check eg log importer can replace a POST request where all parameters are actually in the URL. Also we need to double check that Matomo processes such a tracking request correctly (it should I think).

When enabled, users also need to make sure POST requests are handled correctly by the webserver. This should not be a problem.

Drawback: sendBeacon has no callback to check whether the request was successful or not. This means the callback parameter won't work if someone is using the tracking API.

Universal GA supports sendBeacon since 2014 and users can opt in to it: https://www.thyngster.com/google-analytics-added-sendbeacon-functionality-universal-analytics-javascript-api/