@toredash opened this Issue on November 21st 2019 Contributor

I believe there is an issue with how ping requests are processed.

According to the source visit_last_action_time holds the visit´s end time:

/**
 * This dimension holds the best guess for a visit's end time. It is set the last action
 * time for each visit. `ping=1` requests can be sent to update the dimension value so
 * it can be a more accurate guess of the time the visitor spent on the site.
 *
 * Note: though it is named 'visit last action time' it actually refers to the visit's last action's
 * end time.
 */

https://github.com/matomo-org/matomo/blob/6b4cd1676d524641167ba32776a999e3e37a6a2a/plugins/CoreHome/Columns/VisitLastActionMinute.php

Looking at how the PingRequestProcessor (https://github.com/matomo-org/matomo/blob/3.x-dev/plugins/Heartbeat/Tracker/PingRequestProcessor.php) handles hearbeats, it seems it does not update log_visit.visit_last_action_time, it only updates log_visit.visit_total_time.

log_visit.visit_last_action_time is only updated if an action is sent. I think the issue is that PingRequestProcessor sends an empty Actions:
https://github.com/matomo-org/matomo/blob/2.16.0-b6/plugins/Heartbeat/Tracker/PingRequestProcessor.php#L24-L29

When viewing a users visits, the "x actions in " is correct, as it uses log_visit.visit_first_action_time + log_visit. visit_total_time.

Example of this:
image

For us, this becomes an issue an issue when using CustomReports and Server Time - Hour/Minute (End Of Visit), as it using log_visit.visit_last_action_time (https://github.com/matomo-org/matomo/blob/3.x-dev/plugins/CoreHome/Columns/VisitLastActionMinute.php#L32) to determine the visis end time. This occurs since ping requests does not update log_visit.visit_last_action_time. With the current code, it is not possible to determine when a visit ended for SPA apps, since it seems only an action can that trigger an update of log_visit.visit_last_action_time.

I'm not sure if this is a bug in the PingRequestProcessor, or an issue with how VisitLastActionMinute is expected to compute the visits end time.

Issues i think are relevant:
#9626 #9748

@tsteur commented on November 21st 2019 Member

This is correct and expected behaviour. You can see a bit more info here: https://github.com/matomo-org/matomo/issues/15049#issuecomment-549203055

We'll need to update the docs that it does not extend the visit. You'd need some custom plugin in this case to get this behaviour.

@toredash commented on November 21st 2019 Contributor

@tsteur do you have suggestions on workaround for this usecase ?

Initial thought is to introduce ping=2. If set, it should update log_visit.visit_total_time and log_visit.visit_last_action_time.

Adding a regular _paq.push(['trackEvent', 'ping', 'pong', 'pung']) instead of heartbeats would increase the tablesize for log_link_visit_action, which is not a good idea.

ping=2 seems like, for us at least, a viable option. But if upstream would approve of such change is another story. Or another solutions is to fork the HearBeat plugin, and extend the log_visit table with e.g. last_heartbeat_time which holds last DATETIME for that visit.

In reality, this value is already present since log_visit.visit_last_action_time + log_visit. visit_total_time would equal to last_heartbeat_time

Happy to create an PR if any of the initial thoughts seems like a good idea.

@tsteur commented on November 21st 2019 Member

I suppose you want to extend all visits?

You could do this in a custom plugin without changing the ping value. All you would need to do is put a file like this in Tracker/MyRequestProcessor.php in the plugin which does say

namespace ...
class MyRequestProcessor{

    public function onExistingVisit(&$valuesToUpdate, VisitProperties $visitProperties, Request $request)
    {
        $valuesToUpdate['visit_last_action_time'] = $request->getCurrentTimestamp()
    }
}

Haven't tested it but something like this should work.

@tsteur commented on November 21st 2019 Member

Actually @toredash not sure if we have any public documentation that heart beat extends the visit? it looks like we can actually close the issue.

@toredash commented on November 22nd 2019 Contributor

@tsteur For what I've seen regarding docs, it is for me difficult to say if the docs are explicit enough to answer that question.

https://matomo.org/faq/how-to/faq_21824/

By default, Matomo (Piwik) will accurately track the time spent on your pages for all pages, except the last page view of the visit. By default, on the last page view of any visit, Matomo counts the “Time spent on page” as 0 second. And when your visitor views only one page in the website, the visit duration will be set to a default of 10 seconds.

-> It is possible to configure Matomo to track the time spent on the page accurately, including the last page view of each visit. Learn more about Accurately measure the time spent on each page in the JavaScript Tracking code guide.

I'm not sure if the current implementation does this the way I interpret the docs. From the screenshot in my first post, the ping updates the "Time spent on page" for the last action, not the page itself.

Our applications are primarily single-page-applications, and in rare-cases we might have no actions sent from the client, so we are unable have an accurate report with CustomReports which informs us when the visit ended. This information is used to graph the number of concurrent users active on our service by the minute.

We are able to determine when they came (visit_first_action_time), how long they stayed (visit_total_time), but visit_last_action_time is not a good representation for when the visit ended, even if the source code indicates it:
https://github.com/matomo-org/matomo/blob/48d377071a4c543817c5909c7e95d733b990d8cf/plugins/CoreHome/Columns/VisitLastActionMinute.php#L18-L25

If anything should be changed, it should be the definition of $segmentName = 'visitEndServer*';, since it is labeled as "End Of Visit", which is not correct, it is actually "Time of last action" (https://github.com/matomo-org/matomo/blob/1a76568aa7f85920e8f4b58e0aa2fd32f64de8ad/plugins/VisitTime/lang/en.json):
image

All the information is present in log_visit table, it could be computed as such:

SELECT *, (visit_last_action_time + visit_total_time) AS visit_last_heartbeat
FROM log_visit

I looked at the Segment docs (https://developer.matomo.org/api-reference/Piwik/Plugin/Segment#setsqlsegment) to see if there is a way create a new segment, but I think that performance would suffer it it needs to calculate a value based on two other values.

@toredash commented on November 22nd 2019 Contributor

I suppose you want to extend all visits?

Yes, that is correct, I'll try the code you suggested, thanks!

You could do this in a custom plugin without changing the ping value. All you would need to do is put a file like this in Tracker/MyRequestProcessor.php in the plugin which does say

namespace ...
class MyRequestProcessor{

    public function onExistingVisit(&$valuesToUpdate, VisitProperties $visitProperties, Request $request)
    {
        $valuesToUpdate['visit_last_action_time'] = $request->getCurrentTimestamp()
    }
}

Haven't tested it but something like this should work.

@tsteur commented on November 24th 2019 Member

If anything should be changed, it should be the definition of $segmentName = 'visitEndServer*';, since it is labeled as "End Of Visit", which is not correct, it is actually "Time of last action"

I reckon that makes sense 👍 Any thoughts @mattab ?

@mattab commented on November 25th 2019 Member

Sounds like a good fix to rename the dimension + fix the code comment @tsteur @toredash

I looked at the Segment docs (https://developer.matomo.org/api-reference/Piwik/Plugin/Segment#setsqlsegment) to see if there is a way create a new segment, but I think that performance would suffer it it needs to calculate a value based on two other values.

@toredash I reckon performance of using 2 fields VS 1 field shouldn't be an issue? if you can try this and add a new segment, that would be great... so we can cover all use cases including yours :+1:

@toredash commented on November 25th 2019 Contributor

@mattab

Our use case is that we want to graph the number of unique visitors on a granularity of per minute. That is not possible with how Matomo works now, as the segment only considers when the first and last actions was done, not anything in between.

When using Custom Reports, we have two reports, one with "Server Time - Start of Visit (Hour)" and "Server Time - Start of Visit (Minute)". The other report contains "Server Time - End of Visit (Hour)" and "Server Time - End of Visit (Minute)". This does not give us the correct data, example:

User A join at UNIX time: 1000, and is active to time 2000. visit_first_action_time = 1000, visit_last_action_time = 2000.

User B join at UNIX time: 1500, and is active to time 2500. visit_first_action_time = 1500, visit_last_action_time = 2500.

User A would then be considered to start the visit at hour 00 and minute 16.
User B would then be considered to start the visit at hour 00 and minute 25.

User A would then be considered to end the visit at hour 00 and minute 33.
User B would then be considered to end the visit at hour 00 and minute 41.

When the use the start of visit-report and substract the corresponding values for the same time from the end of visit-report.
We then want to be able to get data that looks like so:

HH:MM - Uniq visitors
00:15 - 0
00:16 - 1
00:17 - 1
...
00:24 - 1
00:25 - 2
00:26 - 2
...
00:32 - 2
00:33 - 1
00:34 - 1
00:35 - 1
...
00:40 - 1
00:41 - 1
00:42 - 0
00:43 - 0

But that is not what we get, we get:

HH:MM - Uniq visitors
00:15 - null
00:16 - 1
00:17 - null
...
00:24 - null
00:25 - 1
00:26 - null
...
00:32 - null
00:33 - null
00:34 - null
00:35 - null
...
00:40 - null
00:41 - null
00:42 - null
00:43 - null

This stems from the fact that the segment only uses the visit_first_action_time and visit_last_action_time, it does not know of the values in between. So we know when the visit started, when it ended, but the reporting API / Custom Report is not able to provide data which states that the visit was active at time (HH:MM) 00:00, 00:01, 00:02, ... .

I know we could solve this with a recursive CTE in SQL, but it seems we are pushing matomo in a direction that is not meant for. Or, we are doing something wrong with CustomReports.

Here are the Start and End reports we are using:
image

image

@mattab We have an ongoing e-mail discussion with you about this in email, if you believe this should be possible in Matomo, should we continue the discussion there ?

@tsteur commented on November 25th 2019 Member

Be good to follow up in the email discussion. @toredash

So in this issue it's only about renaming the segment/dimension.

@toredash commented on December 4th 2019 Contributor

Be good to follow up in the email discussion. @toredash

So in this issue it's only about renaming the segment/dimension.

Yes, that is correct.

Powered by GitHub Issue Mirror