Adding ping=1 tracker query param to allow JS tracker to update the end of visit time for more accurate visit length #8069

diosmosis · 2015-06-09T04:18:18Z

Includes JavaScript tracker addition that sends a 'ping=1' request to Piwik every N seconds for a more accurate visit time, and the Piwik tracker code to handle those requests.

Changes to the JavaScript Tracker

Modified existing 'heart beat' code so it would be more efficient and more accurate. The system as it functions now is as follows:

the setHeartBeatTimer() method is used to configure the heart beat interval.
a page is tracked, which sets the first heart beat timeout
after N seconds pass, the timeout executes. It checks if there has been non-ping tracker activity in the last N seconds, and if not, sends a ping request and schedules a timeout for the next N seconds. If there has been activity, the next timeout is scheduled to be executed N seconds from the last request.

If a window loses focus, the timeout is cleared. When the window gains focus, the timeout is reinstated.

Changes to Piwik tracking code

The change to the tracker is simple, if a ping request is sent and the visit is not new, the action is set to null and only the visit is updated.

Other Changes

I added the ability to issue ping requests to the PHP tracker mainly for testing here: https://github.com/piwik/piwik-php-tracker/tree/ping_addition Must be merged before/after this is merged.
I added the Q promise library for the JS tracker tests. It allowed me to cleanly test the heartbeat code. Without it, the timeout could not be tested, since the JS tests used a wait() function that spun (so the event queue never advanced).

Manual Testing Strategy

For each browser:

start watching visitor log
visit page (assert visitor log has 1 action)
do nothing + wait for 10 secs (assert visitor log has one action, but longer visit length)
do nothing + wait for 10 secs (same assert)
switch tab, wait for 10 secs (assert no change)
wait 10 secs, switch back to tab (assert visitor log has one action, but longer visit length)
do nothing + wait for 10 secs (assert visitor log has one action, but longer visit length)

TODO:

Refs #2041

diosmosis · 2015-06-16T04:01:03Z

Still have to do manual tests & get JSLint test to pass, but this is ready to be reviewed. Will update the description w/ more info soon, but if you want to review now, going commit by commit will help, as I tried to make the commit messages descriptive.

tsteur · 2015-06-23T05:58:39Z

js/piwik.js

                 */
-                setHeartBeatTimer: function (minimumVisitLength, heartBeatDelay) {
-                    var now = new Date();
+                setHeartBeatTimer: function (heartBeatDelay) {


Does this break API? Removing the first parameter? I reckon it was not much used before? Should we mention it in changelog? Or maybe we do already? :)

It might break code that uses it, however, it didn't actually do anything before. It was incomplete, just half the feature. It sent useless requests, probably creating new actions, no one should be using it.

Probably not everyone likes it, I'd personally name it heartBeatDelayInSeconds and also mention it in doc block that it is seconds since one is used to deal with ms eg in setTimeout etc.

tsteur · 2015-06-23T06:21:52Z

I'll have another look tomorrow and not only have a look at the diff... so far looks like a nice, small addition

tsteur · 2015-06-23T23:14:39Z

js/piwik.js

+                    heartBeatDown();
+                });
+
+                heartBeatUp();


When looking at the code I found it a bit confusing that setUpHeartBeat() does also directly sends or sets a timeout for heartBeatUp although it's convenient. I'd maybe remove it from there and call it separately but up to you

I don't really understand this, setUpHeartBeat() sets up events and the first timeout, but the timeout is set via heartBeatUp(). heartBeatUp() is called in many places, including in heartBeatUp itself, because the timeout does not always exist even if the heart beat is enabled.

I will probably merge w/o this change, but if you want to clarify we can talk on slack.

tsteur · 2015-06-23T23:25:55Z

FYI: The minified piwik.js version needs to be created

…s misleading. Though it is initially set to the start time of a visit's last action, it is supposed to represent the visit's actual end time.

… is more performant and executes much less code on user websites. New piwik.js tests included as well as an additional utility to make writing future tests easier (use jquery to parse tracker requests that are returned from fetchTrackedRequests). However tests don't currently pass since qunit is not super helpful for testing async code.

…he piwik.js test).

…eature in the JS tracker.

…me, since we should ping N seconds after last ping also. And if tracking request is sent before heart beat timer is configured, setup heart beat when configuring heart beat timer.

…nableHeartBeatTimer, rename clearHeartBeat to disableHeartBeatTimer, make sure the timer will be set up after any tracker request, not just page view.

…elay cannot be lower than 1 second.

…doesn't appear in minified piwik.js. And set last tracker request time in makeSureThereIsAGapAfterFirstTrackingRequestToPreventMultipleVisitorCreation for increased code clarity + so it will be set when bulk requests sent.

… to ...InSeconds since its in milliseconds.

Adding ping=1 tracker query param to allow JS tracker to update the end of visit time for more accurate visit length.

diosmosis · 2015-06-25T12:18:20Z

Moved the interval changing code to here: #8213

…matomo-org/matomo#2041

matomo-org/matomo#8069 matomo-org/matomo#2041

diosmosis self-assigned this Jun 9, 2015

diosmosis added this to the 2.14.0 milestone Jun 9, 2015

diosmosis force-pushed the 2041_visitor_ping branch 6 times, most recently from 7f1ef90 to 261847e Compare June 13, 2015 05:06

diosmosis added Needs Review PRs that need a code review and removed Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. labels Jun 16, 2015

diosmosis force-pushed the 2041_visitor_ping branch from 261847e to 06cc452 Compare June 16, 2015 04:57

tsteur reviewed Jun 23, 2015
View reviewed changes

diosmosis force-pushed the 2041_visitor_ping branch from be10196 to c5ef759 Compare June 23, 2015 11:14

tsteur reviewed Jun 23, 2015
View reviewed changes

diosmosis added 12 commits June 25, 2015 04:13

Add documentation for VisitLastActionTime dimension since it's name i…

9bed880

…s misleading. Though it is initially set to the start time of a visit's last action, it is supposed to represent the visit's actual end time.

Adding Q JavaScript promise library for use in tests (specifically, t…

2a0a534

…he piwik.js test).

Use Q promise library to accurately and cleanly test the heart beat f…

aee6c74

…eature in the JS tracker.

Fix JSLint errors by moving functions around and making other changes.

89c5e31

Adding new heart beat / ping feature to CHANGELOG.md.

1deea27

Do not ignore ping requests when checking the last tracker request ti…

b2667ba

…me, since we should ping N seconds after last ping also. And if tracking request is sent before heart beat timer is configured, setup heart beat when configuring heart beat timer.

Clean up some additions in this branch, rename setHeartBeatTimer to e…

3a40e6c

…nableHeartBeatTimer, rename clearHeartBeat to disableHeartBeatTimer, make sure the timer will be set up after any tracker request, not just page view.

Add note about Q promise library to the CHANGELOG.

454d27e

Clear up heart beat code in piwik.js a bit and make sure heart beat d…

06f46e3

…elay cannot be lower than 1 second.

Update minified piwik.js file.

8b4a1c7

diosmosis force-pushed the 2041_visitor_ping branch from ce5f651 to 8b4a1c7 Compare June 25, 2015 11:16

diosmosis added 3 commits June 25, 2015 04:18

Fix configHeartBeatDelay rename, shouldn't have renamed that variable…

0bbf109

… to ...InSeconds since its in milliseconds.

Remoe heart beat events in disableHeartBeatTimer in JS tracker.

ced91af

Minify piwik.js file.

e3d7c6c

diosmosis added a commit that referenced this pull request Jun 25, 2015

Merge pull request #8069 from piwik/2041_visitor_ping

ea8935d

Adding ping=1 tracker query param to allow JS tracker to update the end of visit time for more accurate visit length.

diosmosis merged commit ea8935d into master Jun 25, 2015

diosmosis deleted the 2041_visitor_ping branch June 25, 2015 12:12

diosmosis mentioned this pull request Jun 25, 2015

Allow JS tracker to change heart beat wait interval exponentially #8213

Closed

diosmosis mentioned this pull request Jun 25, 2015

Support Ping requests to report a better 'time on page' #2041

Closed

diosmosis pushed a commit that referenced this pull request Jun 25, 2015

Refs #8069, update PHP tracker submodule.

f7b5a5b

tsteur mentioned this pull request Jun 25, 2015

Make sure ping=1 does not create duplicated actions #8218

Closed

This was referenced Jun 26, 2015

Enable by default the Heart beat timer to process a more accurate time on page and visit duration #8225

Open

Ping=1 requests tweaks #8223

Merged

mattab pushed a commit to matomo-org/developer-documentation that referenced this pull request Jun 26, 2015

Add doc for enableHeartBeatTimer function. refs matomo-org/matomo#8069 …

2d9fa99

…matomo-org/matomo#2041

mattab pushed a commit to matomo-org/developer-documentation that referenced this pull request Dec 18, 2015

Add documentation for enableHeartBeatTimer

92ed14b

matomo-org/matomo#8069 matomo-org/matomo#2041

mattab mentioned this pull request Dec 18, 2015

Add documentation for enableHeartBeatTimer matomo-org/developer-documentation#114

Merged

dbakker mentioned this pull request Jan 5, 2016

Document heartbeat/"time on page" information for the HTTP Tracking API #9477

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding ping=1 tracker query param to allow JS tracker to update the end of visit time for more accurate visit length #8069

Adding ping=1 tracker query param to allow JS tracker to update the end of visit time for more accurate visit length #8069

diosmosis commented Jun 9, 2015

diosmosis commented Jun 16, 2015

tsteur Jun 23, 2015

diosmosis Jun 23, 2015

tsteur Jun 23, 2015

tsteur commented Jun 23, 2015

tsteur Jun 23, 2015

diosmosis Jun 24, 2015

tsteur commented Jun 23, 2015

diosmosis commented Jun 25, 2015

Adding ping=1 tracker query param to allow JS tracker to update the end of visit time for more accurate visit length #8069

Adding ping=1 tracker query param to allow JS tracker to update the end of visit time for more accurate visit length #8069

Conversation

diosmosis commented Jun 9, 2015

Changes to the JavaScript Tracker

Changes to Piwik tracking code

Other Changes

Manual Testing Strategy

diosmosis commented Jun 16, 2015

tsteur Jun 23, 2015

Choose a reason for hiding this comment

diosmosis Jun 23, 2015

Choose a reason for hiding this comment

tsteur Jun 23, 2015

Choose a reason for hiding this comment

tsteur commented Jun 23, 2015

tsteur Jun 23, 2015

Choose a reason for hiding this comment

diosmosis Jun 24, 2015

Choose a reason for hiding this comment

tsteur commented Jun 23, 2015

diosmosis commented Jun 25, 2015