Hi @asharov

To loop over the last visits details API I would recommend not using filter_offset but minTimestamp.

Eg

?module=API&method=Live.getLastVisitsDetails&idSite=62&date=2020-09-02,2020-09-03&period=range&format=json&filter_limit=1000

Then you get the timestamp of the last visit and fetch all following visits from the timestamp after that.

?module=API&method=Live.getLastVisitsDetails&idSite=62&date=2020-09-02,2020-09-03&period=range&format=json&filter_limit=1000&minTimestamp=XYZ

And you repeat the same until there are less than 1000 results.

There could be an edge case where there are many visits for the very same second and only some of them are returned. So it may pay off to do subtract 1s from the minTimestamp to fetch the visits from the last second again and discard them from the previous response.

Hope this helps.

Hi @tsteur

Thanks for the suggestion. Unfortunately I cannot make it work. The problem is that the returned visits are ordered latest one first. So when I make the request

https://demo.matomo.org/?module=API&method=Live.getLastVisitsDetails&idSite=62&date=2020-09-02,2020-09-03&period=range&format=json&filter_limit=1000

the latest result is from "2020-09-03 23:56:08", that is, from the very end of the date range I requested. If I use the timestamp, 1599170168, from this minus 1s as minTimestamp, like this

https://demo.matomo.org/?module=API&method=Live.getLastVisitsDetails&idSite=62&date=2020-09-02,2020-09-03&period=range&format=json&filter_limit=1000&minTimestamp=1599170167

I get only one response. As expected, I guess, since the timestamp is near the end of the range. Also, trying to set minTimestamp to an early part of the date range, like this

https://demo.matomo.org/?module=API&method=Live.getLastVisitsDetails&idSite=62&date=2020-09-02,2020-09-03&period=range&format=json&filter_limit=1000&minTimestamp=1599000000

gives the same response as the request without minTimestamp.

Looking at the API documentation, I noticed the filter_sort_order and filter_sort_column parameters, but they also behave strangely. When I add filter_sort_order=asc&filter_sort_column=serverTimestamp to my request, the responses do come in ascending order by time, but they start from the first visit of 2020-09-03, and not from the first visit of 2020-09-02, which is the start of the queried date range. It doesn't seem possible to get the visits from 2020-09-02 at all with these parameters.

Do you have any further suggestions of what could work?

Best,
Jaakko

Thanks for the detailed information @asharov

Looking through the code I think this is a regression from #14700 as filter_sort_order is no longer working as expected. It would work if filter_sort_order was working correctly.

In particular, I think this is causing the problem: https://github.com/matomo-org/matomo/pull/14700/files#diff-366368dca569c3a4e0b3b0dc3724ecb0R115-R163

Say the requested date range is 2020-08-02,2020-08-30. Then it would apply the filter_sort_order but it would always first look for visits in 2020-08-30, then in the next query in 2020-08-22,2020-08-29, then 2020-08-02,2020-08-21. It should do it in this case probably do it the other way around.

Does the logic work when you request the data for each day individually (with filter_sort_order parameter)?

Yes, if I request only for a single day, using filter_sort_order and looping with minTimestamp as you suggested does seem to get all the visits for that day. That is, with the initial request

https://demo.matomo.org/?module=API&method=Live.getLastVisitsDetails&idSite=62&date=2020-09-03&period=day&format=json&filter_limit=1000&filter_sort_order=asc&filter_sort_column=serverTimestamp

and then making further requests by setting minTimestamp based on the serverTimestamp of the last visit from the previous response. The first and last times of the overall visits are at the ends of the day and there are no gaps between different responses.

So I think I can use this method in our case now to patch any missing data as a workaround, but it would of course be preferable to have this method working over a range of days.

yes definitely it should work over a range of days. It's a regression as it used to work and a fix is likely not too hard.

Hey @tsteur! I would like to take care of this. Would simply reversing the order of the return array of splitDatesIntoMultipleQueries as in
$queries = $this->splitDatesIntoMultipleQueries($dateStart, $dateEnd, $limit, $offset);
be enough (of course, provided that $filterSortOrder === 'asc')? Can you foresee further issues with this change? :)

@TheCrowned this should work. Just need to make sure that in each date combination in the array the first one is the lower date and the second the higher date. Eg by default it would maybe return something like

[
 ['2020-09-28 00:00:00', '2020-09-28 23:59:59'], // look first last24 hours
 ['2020-09-21 00:00:00', '2020-09-27 23:59:59'] // then last 6 or 7 days
 ['2020-08-21 00:00:00', '2020-09-20 23:59:59'] // then last 30 days or so
]

it should then return

[
 ['2020-08-21 00:00:00', '2020-08-21 23:59:59'], // look first last24 hours
 ['2020-08-22 00:00:00', '2020-08-28 23:59:59'] // then last 7 days
 ['2020-08-29 00:00:00', '2020-09-28 23:59:59'] // then last 30 days or so
]

It's just an example. I'm basically only meaning it's important it returns eg ['2020-08-21 00:00:00', '2020-08-21 23:59:59'] and not the other way around ['2020-08-21 23:59:59', '2020-08-21 00:00:00'].

It should not simply reverse the returned order from the initial return. It should not return this:

[
 ['2020-08-21 00:00:00', '2020-09-20 23:59:59'] // then last 30 days or so
 ['2020-09-21 00:00:00', '2020-09-27 23:59:59'] // then last 7 days
 ['2020-09-28 00:00:00', '2020-09-28 23:59:59'], // look first last24 hours
]

I haven't looked into the implementation so not sure how it be best developed or what needs to be done. We basically always want to make sure to first look only at one day, then one week, then 30 days, etc for performance reasons.