for 2)
#9505

for 1)
nice: lots of people stays that long :-)

demo seems to be running v2.16b1

Thx for that, looks like something we should look into on DB level etc. @mattab

hmm first thought the only reason can be
"heart beat also working when page is in background in browsers with several tabs opened",
than I had a look again:
the order of visits is messed up too (shown times are not chronological)
=> maybe one is somehow dealing with local times of visitors??
or one take start end from different visitors???
please see attachment

pretty strange

another strange finding:
looking in the sourcecode of tracked website in demo, heartbeat is enable but no special period is set.
With this it should be 15 seconds.
When looking at the shown durations of visits with only one action (this is where heartbeat helps determine time on page)

• these are not always multiples of 15 seconds as expected and
• these are also not multiples of 15 second plus typical loading times (1-4seconds) as this would be the other expected way to work

just found the date when strange things started:

btw:
are there already any checks available to automatic catch huge changes like that?

would be good to have e.g. custom alerts defined for things like this in piwik's demo and keep on deploying latest betas to demo.

hmm can someone give me a hint how to define an alert for a huge change of "average visit duration" within custom alerts plugin?

also new visits from today are tracked as too long (and are shuffled in ordered)
imho this has two effects:
it's not the best advertising seeing this in a demo
and on the other hand some statistics becomes tainted.
=> One should think of fixing this with high prio
or separate "public demo" from trial and error with betas.

FYI: I had a look in the database and in the access logs for one visitor and eg the total time of 7h 50minutes was actually correct for that user. There were thousands of ping requests over several hours for that visitor and also the database entries seem valid.

I tested manually whether it stops tracking when the tab is no longer focused / opened and it worked.

@hpvd if you can reproduce an issue that PIwik is over-tracking, let us know the steps / browsers to use to reproduce. our tests show that it "should" work as expected (track only in active tab). It looks like a lot of users may open a forum thread and leave it there open, for hours!

Maybe in our ping mechanism we should check that the mouse or some keyboard keys are actually used, before sending a ping? maybe there is no advantage to send a ping if keyboard/mouse are idle...

Also I presume most useful feature would be to really increase ping interval over time as I had eg like 4000 ping requests for one page for the visitor that I analyzed which might not be needed to be that much :) see #9423

many thanks for looking at this.
If that many people have pages opened and keep them active in foreground there is something wrong - how many monitors they have? do they not have to work? or look at facebook in between? ;-)

Just digging a little deeper:

First shot-> first result:
This way one can easily reproduce that foreground detection is not always working:
going to:
http://demo.piwik.org/index.php?module=CoreHome&action=index&idSite=7&period=day&date=2016-01-15#?module=Live&action=indexVisitorLog&idSite=7&period=day&date=2016-01-15
right mouse button click on one of the links to pages people visit for a long time
e.g. http://forum.piwik.org/t/piwik-not-showing-accurate-stats/7684
and " open in new tab"
=> page opens in background and visit time is counting all day long (but page has never been seen by the user -> it's only opend to maybe read it later... maybe...)

this is from testing with opera:

same in chrome:

still counting and sorted to top of list:

• browser was most of the time in backgrund (yes I have to do some other things...)
• browser tab was always in background

but still counting

=> sensing if there is any mouse or keyboard movements in tracked browser tab before sending each ping seems to be an abolute must
edit:
maybe before each ping one can simply check if mouse position in browser window is the same than before last ping - if so no new ping is send.

@tsteur
Also I presume most useful feature would be to really increase ping interval over time as I had eg like 4000 ping requests for one page for the visitor that I analyzed which might not be needed to be that much :) see #9423

sure ;-), another idea: clean it afterwards #9530

just brought the tab to foreground for some time and closed it afterwards
-> (as expected) nothing of this could be seen in tracking
=> so we do not know if the visitor has seen the page

maybe a possible easy first solution for tracking correct visit-time:
before each ping one can simply check if mouse position in browser window is still the same than before last ping - if so no new ping is send.

Maybe this helps to make a very first try:
https://stackoverflow.com/questions/7790725/javascript-track-mouse-position

You're right! See comment #9504 (comment) and following. When opening browser tab in new window with eg right click and even though the tab might have never been opened we do send ping requests.

We should possibly aim to work on this for 2.16.0

good to hear!
If there is a first solution, please put it to demo where we had reproduced this problem (today it's still 2.16b1 with 100+ commits to master and b3 available)
=> so we can easily see within same environment if it helps to get more plausible visiting times.
Otherwise we have to look for other causes for this long visit times...

Hi @hpvd demo was updated with our latest beta

just to show the actual progress:
It's already getting more plausible and less strange again,
but average time on page is more than 2-3times as long as it was before heartbeat function was activated
and it's also not that constant in average as it was...

=> what do we expect to see here?
see (2) in #8225 (comment)
=> What would be "good enough"?

to see the results of made changes faster by asap adaption of usage by all visitors,
a versioning of piwik's js to bypass visitor's browser caches maybe the way to go
see #9619

Hi @hpvd we've released 2.16.0-RC1 and deployed on demo - how does it look to you?

Hi @mattab
many thanks! We need a little time to have some visitors using it...
btw: Which values would you expect to see in avarage time with now having heart beat enabled?
I'm not really sure about this.
maybe a bit longer because last page visited adds additional time?
or a little shorter because now also one page visits have a time on page which should be considered within calculation of average time? (are they? Or were they even before with 0s?)
or because of both: nearly the same? e.g. +-20% ?

hmmm the only thing which would not look very plausible to me is a jump of +-100% in average

please have a look on new #9645 which may influence avarage time too...

The times on page are expected to be pretty much the same as without heartbeat apart from the last action. The last action should have a longer time on page. With the current implementation it is expected that it should be max around 30 minutes for the last action (depends on a config entry and the type of the last action). How much longer the times are depends on the users and their behaviour.

ok, that's the same I would expect.
With RC1 there is still more than factor 3 difference in average visit duration between without and with heartbeat (looks the same it was with b6)
Does anyone has any ideas what may be the reason(s)?

Average time before heart beat was about 2 minutes. When visitors leave tab open for a while at the end of their visit it might be ok that it increases to say 6 minutes? (Also again piwik.js might be still cached for some users I haven't checked).

It would be interesting to see what the median value was as this one might be more accurate here. When having a low visit duration of say less than 3 minutes which is probably the case for most websites a few users can increase the avg visit duration a lot just by having the tab open for 30 minutes. This might be less of a problem the more visits there are. Would be neat to offer users a median graph as well :)

Also in this case the forum has many visits with only one pageview. Meaning the avg visit duration was rather very low for many visits. Now we are actually able to track the duration also for just one pageview so it might be expected to have 3 times higher avg visit duration.

yes median graph would be interesting.

But shouldn't the "long open last pages of visits" be compensated by the now tracked "only one page visits" (which have a short visit time in total)?

or were this "only one page visits" put into caculation with 0s in the past ?
(because we couldn't track their duration without heartbeat)

or were this "only one page visits" put into caculation with 0s in the past ?

I presume so

Hi @hpvd if you find any issue with heartbeat logic or reporting, please create new issue, cheers