Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Chrome's "Data Saver" mode prevents tracking of many visitors using Chrome browser #7733

Closed
ericimprint opened this issue Apr 21, 2015 · 20 comments
Closed

Chrome's "Data Saver" mode prevents tracking of many visitors using Chrome browser #7733

ericimprint opened this issue Apr 21, 2015 · 20 comments
Assignees
@tsteur
Labels
Bug For errors / faults / flaws / inconsistencies etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Milestone
2.15.0

Comments

@ericimprint
Copy link

Visitors using chrome's Data Saver mode are not tracked. I can see them in Google Analytics, but not in Piwik.
@ericimprint ericimprint changed the title Chrome's "Data Saver" mode not tracking Chrome's "Data Saver" mode prevents tracking Apr 21, 2015
@mattab
Copy link
Member

mattab commented Jul 14, 2015

Hi @ericimprint

Thanks for the report. In Chrome documentation they explain how to disable the feature, maybe this is what we could do:

As a site owner, how do I opt-out from content optimization?

Data compression proxy respects the standard Cache-Control: no-transform directive. Site owners can mark individual resources with this directive and the proxy will pass them through directly to the browser.

@mattab mattab added this to the Mid term milestone Jul 14, 2015
@mattab
Copy link
Member

mattab commented Jul 14, 2015

I haven't checked where this should be done, but I guess maybe chrome won't load piwik.js in some cases, and we would need to return this Cache-Control: no-transform for piwik.js?

@mattab mattab removed this from the Mid term milestone Jul 14, 2015
@ericimprint
Copy link
Author

I added Cache-Control: no-transform to piwik.js via htaccess and it did not change anything. I could not see my visit from my phone with data saver turned on. Also, analytics.js does not have that set (not that Google wouldn't have already worked away around that internally).

@mattab mattab added the Bug For errors / faults / flaws / inconsistencies etc. label Jul 15, 2015
@mattab
Copy link
Member

mattab commented Jul 15, 2015

I added Cache-Control: no-transform to piwik.js via htaccess and it did not change anything.

If you open Chrome > Developer tools, do you see the piwik.js being loaded in your browser?

@mattab mattab added this to the 2.15.0 milestone Jul 15, 2015
@ericimprint
Copy link
Author

Piwik.js is being loaded both when the tracking is working and when it is not. Watching the network waterfall I can see it come up both when data saver is on and when it is off.

@tsteur
Copy link
Member

tsteur commented Aug 7, 2015

FYI: In the tracker debug log I get:

Search bot detected, visit excluded
Visit is already excluded, no need to check DoNotTrack support.

Here's some more info from $_SERVER (I removed some keys and some information):

array(73) {
  string(11) "en_US.UTF-8"
  ["LC_ALL"]=>
  string(11) "en_US.UTF-8"
  ["ORIG_PATH_INFO"]=>
  string(10) "/piwik.php"
  ["SCRIPT_NAME"]=>
  string(10) "/piwik.php"
  ["REQUEST_URI"]=>
  string(498) "/piwik.php?action_name=Testimonials%20-%20Analytics%20Platform%20-%20Piwik&idsite=1&rec=1&r=592202&h=10&m=49&s=38&url=http%3A%2F%2Fpiwik.org%2Ftestimonials%2F&urlref=http%3A%2F%2Fpiwik.org%2Fhosting%2F&_id=8ef979f20d4f641e&_idts=1433737138&_idvc=29&_idn=0&_refts=1438936067&_viewts=1438847822&_ref=https%3A%2F%2Fwww.google.de%2F&send_image=0&pdf=1&qt=0&realp=0&wma=0&dir=0&fla=1&java=1&gears=0&ag=0&cookie=1&res=1280x800&_cvar=%7B%221%22%3A%5B%22Domain%20landed%22%2C%22piwik.org%22%5D%7D&gt_ms=379"
  ["QUERY_STRING"]=>
  string(487) "action_name=Testimonials%20-%20Analytics%20Platform%20-%20Piwik&idsite=1&rec=1&r=592202&h=10&m=49&s=38&url=http%3A%2F%2Fpiwik.org%2Ftestimonials%2F&urlref=http%3A%2F%2Fpiwik.org%2Fhosting%2F&_id=8ef979f20d4f641e&_idts=1433737138&_idvc=29&_idn=0&_refts=1438936067&_viewts=1438847822&_ref=https%3A%2F%2Fwww.google.de%2F&send_image=0&pdf=1&qt=0&realp=0&wma=0&dir=0&fla=1&java=1&gears=0&ag=0&cookie=1&res=1280x800&_cvar=%7B%221%22%3A%5B%22Domain%20landed%22%2C%22piwik.org%22%5D%7D&gt_ms=379"
  ["REQUEST_METHOD"]=>
  string(3) "GET"
  ["SERVER_PROTOCOL"]=>
  string(8) "HTTP/1.1"
  ["GATEWAY_INTERFACE"]=>
  string(7) "CGI/1.1"
  ["REDIRECT_URL"]=>
  string(10) "/piwik.php"
  ["REDIRECT_QUERY_STRING"]=>
  string(487) "action_name=Testimonials%20-%20Analytics%20Platform%20-%20Piwik&idsite=1&rec=1&r=592202&h=10&m=49&s=38&url=http%3A%2F%2Fpiwik.org%2Ftestimonials%2F&urlref=http%3A%2F%2Fpiwik.org%2Fhosting%2F&_id=8ef979f20d4f641e&_idts=1433737138&_idvc=29&_idn=0&_refts=1438936067&_viewts=1438847822&_ref=https%3A%2F%2Fwww.google.de%2F&send_image=0&pdf=1&qt=0&realp=0&wma=0&dir=0&fla=1&java=1&gears=0&ag=0&cookie=1&res=1280x800&_cvar=%7B%221%22%3A%5B%22Domain%20landed%22%2C%22piwik.org%22%5D%7D&gt_ms=379"
  ["SERVER_ADMIN"]=>
  string(18) "[no address given]"
  ["REMOTE_ADDR"]=>
  string(13) "66.XXX.XXX.XXX"
  ["SERVER_SIGNATURE"]=>
  string(0) ""
  ["HTTP_X_REAL_IP"]=>
  string(13) "66.XXX.XXX.XXXX"
  ["HTTP_USER_AGENT"]=>
  string(121) "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.130 Safari/537.36"
  ["HTTP_CONNECTION"]=>
  string(5) "close"
  ["HTTP_X_FORWARDED_FOR"]=>
  string(14) "84.XXX.XXX.XXX"
  ["HTTP_VIA"]=>
  string(28) "1.1 Chrome-Compression-Proxy"
  ["HTTP_UPGRADE_INSECURE_REQUESTS"]=>
  string(1) "1"
  ["HTTP_SCHEME"]=>
  string(4) "http"
  ["HTTP_FORWARDED"]=>
  string(18) "for=84.XXX.XXX.XXX"
  ["HTTP_CACHE_CONTROL"]=>
  string(9) "max-age=0"
  ["HTTP_ACCEPT_LANGUAGE"]=>
  string(35) "en-US,en;q=0.8,de-DE;q=0.6,de;q=0.4"
  ["HTTP_ACCEPT_ENCODING"]=>
  string(19) "gzip, deflate, sdch"
  ["HTTP_ACCEPT"]=>
  string(74) "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
  ["GEOIP_ORGANIZATION"]=>
  string(6) "Google"
  ["GEOIP_ISP"]=>
  string(6) "Google"
  ["GEOIP_POSTAL_CODE"]=>
  string(5) "94043"
  ["GEOIP_LONGITUDE"]=>
  string(11) "-XXX.XX7404"
  ["GEOIP_LATITUDE"]=>
  string(9) "XX.XXXXXX"
  ["GEOIP_AREA_CODE"]=>
  string(3) "XXX"
  ["GEOIP_DMA_CODE"]=>
  string(3) "XXX"
  ["GEOIP_CITY"]=>
  string(13) "Mountain View"
  ["GEOIP_REGION"]=>
  string(2) "CA"
  ["GEOIP_COUNTRY_NAME"]=>
  string(13) "United States"
  ["GEOIP_COUNTRY_CODE"]=>
  string(2) "US"
  ["REDIRECT_STATUS"]=>
  string(3) "200"
  ["REDIRECT_GEOIP_ORGANIZATION"]=>
  string(6) "Google"
  ["REDIRECT_GEOIP_ISP"]=>
  string(6) "Google"
  ["REDIRECT_GEOIP_POSTAL_CODE"]=>
  string(5) "94043"
  ["REDIRECT_GEOIP_LONGITUDE"]=>
  string(11) "-XXX.XXXXXX"
  ["REDIRECT_GEOIP_LATITUDE"]=>
  string(9) "XX.XXXXXX"
  ["REDIRECT_GEOIP_AREA_CODE"]=>
  string(3) "XXX"
  ["REDIRECT_GEOIP_DMA_CODE"]=>
  string(3) "XXX"
  ["REDIRECT_GEOIP_CITY"]=>
  string(13) "Mountain View"
  ["REDIRECT_GEOIP_REGION"]=>
  string(2) "CA"
  ["REDIRECT_GEOIP_COUNTRY_NAME"]=>
  string(13) "United States"
  ["REDIRECT_GEOIP_COUNTRY_CODE"]=>
  string(2) "US"
  ["FCGI_ROLE"]=>
  string(9) "RESPONDER"
  ["PHP_SELF"]=>
  string(10) "/piwik.php"
  ["REQUEST_TIME_FLOAT"]=>
  float(1438937793.5144)
  ["REQUEST_TIME"]=>
  int(1438937793)
  ["argv"]=>
  array(1) {
    [0]=>
    string(487) "action_name=Testimonials%20-%20Analytics%20Platform%20-%20Piwik&idsite=1&rec=1&r=592202&h=10&m=49&s=38&url=http%3A%2F%2Fpiwik.org%2Ftestimonials%2F&urlref=http%3A%2F%2Fpiwik.org%2Fhosting%2F&_id=8ef979f20d4f641e&_idts=1433737138&_idvc=29&_idn=0&_refts=1438936067&_viewts=1438847822&_ref=https%3A%2F%2Fwww.google.de%2F&send_image=0&pdf=1&qt=0&realp=0&wma=0&dir=0&fla=1&java=1&gears=0&ag=0&cookie=1&res=1280x800&_cvar=%7B%221%22%3A%5B%22Domain%20landed%22%2C%22piwik.org%22%5D%7D&gt_ms=379"
  }
  ["argc"]=>
  int(1)
}

Interesting could be eg

 ["HTTP_VIA"]=> string(28) "1.1 Chrome-Compression-Proxy"

@tsteur tsteur mentioned this issue Aug 7, 2015
Closed
@tsteur
Copy link
Member

tsteur commented Aug 7, 2015

Created an issue for the device-detector library: matomo-org/device-detector#5363

I presume we should keep this one open?

@mattab
Copy link
Member

mattab commented Aug 7, 2015

I presume we should keep this one open?

@tsteur yes, until the issue in device-detector will be fixed

Good find!

@sgiehl
Copy link
Member

sgiehl commented Aug 8, 2015

As mentioned in the device detector issue:

I assume the problem is within Piwik.
The requests through chromes data saver are coming from a google server. In my test that was: 66.249.93.251

Those IPs are excluded by
https://github.com/piwik/piwik/blob/master/core/Tracker/VisitExcluded.php#L196

@mattab
Copy link
Member

mattab commented Aug 9, 2015

Ok, maybe the solution is to consider Google IP addresses requests as bots only when the HTTP_VIA header does not contain Chrome-Compression-Proxy? (in which case it would not be a non human bot, but it would be a bot-proxying-a-real-visitor)

@tsteur tsteur self-assigned this Aug 10, 2015
@tsteur tsteur mentioned this issue Aug 10, 2015
Merged
@mattab
Copy link
Member

mattab commented Aug 13, 2015

some users sometimes report that Piwik track less data than GA, it could be that this bug fix will improve this situation

@ericimprint
Copy link
Author

Awesome. Thanks guys. How can I test this before the release?

@mattab
Copy link
Member

mattab commented Aug 17, 2015

@ericimprint you can use Piwik 2.15.0-b1 which includes this fix and few others: http://piwik.org/faq/how-to-update/faq_159/

@ericimprint
Copy link
Author

Yesterday before updating to the beta our sites visits were about 77% of GA. Today they are running at 91% of GA. Thanks for the fix.

@mattab
Copy link
Member

mattab commented Aug 18, 2015

Hi @ericimprint that's really awesome to hear. It is a big win for Piwik product and the community at large. Fixing this bug means Piwik became much better at tracking data 🚀

Maybe, we could dig down further and try to explain those 9% difference with GA. What I suggest first would be to look in Piwik and GA and compare for Browser versions report, whether some rows are showing very different "Pageviews" count. Maybe piwik.js has some bug when tracking some particular browser versions? if we find that some browser versions have much less traffic on Piwik than GA, we will have something to investigate further 👍

@Tralapo
Copy link

Tralapo commented Aug 19, 2015

@ericimprint @mattab The difference is indeed big! Some of my numbers:

Past Sunday, without the beta installed:
Piwik: 2997 visitors and 5768 pageviews.
GA: 3727 visitors and 6288 pageviews.

Yesterday, with the beta installed:
Piwik: 2560 visitors, 4441 pageviews
GA: 2619 visitors and 4443 pageviews.

Especially the pageviews are incredibly close! Good job. Visitors are still a bit off, but not by much. But this can be seen throughout the day, the real time visit counter is almost always a bit lower then GA's. Without the beta this could run up to a difference of 30 visitors, now with the beta it's between 0-10.

@Tralapo
Copy link

Tralapo commented Aug 22, 2015

@mattab @ericimprint Although my visitor and pageview counts are much, much closer since installing the beta, like I stated in the comment above, my real-time visit counter started to be very incorrect again.

For example, right now:
GA: 27 visitors at the moment
Piwik: 8 visitors.

Is this normal behaviour for Piwik? Or should I open a new Issue for this.

@ericimprint
Copy link
Author

I have found that to be normal as they define live visitors differently. The main thing you want to look at is total daily pageviews matching. Matt feel free to correct me if I am wrong.

@mattab
Copy link
Member

mattab commented Aug 26, 2015 via email

@Tralapo
Copy link

Tralapo commented Aug 26, 2015

I haven't found what they define as live exactly, but it's quite impressive to be honest.

GA has a Realtime page, where you can see the counter and a table with all the pages that number of visitors is on at the moment. (That would be great to have in Piwik too by the way. The current real-time logbook is to cluttered to get a fast overview)

I did some tests with it and it's quite fast. If I open a tab and navigate to a random page, it pops up in the table (and the counter) within seconds. Same goes for closing the tab, it disappears within seconds. So I don't know how, but they can define live very live.

Did the same tests with Piwik, but it's not that fluid.

@mattab mattab added the Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical. label Oct 13, 2015
@mattab mattab changed the title Chrome's "Data Saver" mode prevents tracking Chrome's "Data Saver" mode prevents tracking of many Chrome visitors Oct 13, 2015
@mattab mattab changed the title Chrome's "Data Saver" mode prevents tracking of many Chrome visitors Chrome's "Data Saver" mode prevents tracking of many visitors using Chrome browser Oct 13, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tsteur tsteur

Labels
Bug For errors / faults / flaws / inconsistencies etc. Major Indicates the severity or impact or benefit of an issue is much higher than normal but not critical.
Projects
None yet
Milestone
2.15.0
Development

No branches or pull requests
5 participants
@tsteur @mattab @sgiehl @Tralapo @ericimprint