@ibril15 opened this Issue on August 18th 2020

Hi All,

I seem to be having the same issue. Premised environment running 3.13.0. For me it seems the issue is when the week goes across months.

For Example:

Date Range: Week From 2020-07-27 To 2020-08-02

If I select Custom Range for 2020-07-27 To 2020-08-01 (one day previous), I get 1,661 visits. If I add 2020-08-02 to the range, or use the pre-built Week option for that week, I get 1,407 visits. On 2020-08-02 I have 314 visits.

If I include 2020-08-02 in a range like 2020-07-28 to 2020-08-03 (doesn't include a whole week), the sum of the individual days equals the number of visits and everything looks right. If I change the range to 2020-07-26 to 2020-08-03 (includes a whole week and some extra days), the numbers are wrong again.

So it appears that the issue may be when calculating a preset range (like "Week"), maybe when the week crosses months? When I select a custom date range that includes a pre-set range like week, I'm assuming the range detects that and uses the pre-archived week. And if that pre-archived week is wrong, all the subsequent calculations that depend on it will also be wrong.

I tried to invalidate the archive and re-archive the data but that didn't help. Please let me know if I can do any further troubleshooting.

Thanks a lot.

-Igor

Originally posted by @ibril15 in https://github.com/matomo-org/matomo/issues/15363#issuecomment-675508797

@tsteur commented on August 18th 2020 Member

Note: sounds like sum issue with date range invalidation.

@tsteur commented on August 18th 2020 Member

@ibril15 do you mind letting us know how you invalidated the archive? Per API call or CLI command? Do you remember which parameters were used?

@ibril15 commented on August 18th 2020

I used the Invalidate Reports plugin (v0.1.2). Selected All Segments, All Historical Data, and just the website with the issue.

@tsteur commented on August 19th 2020 Member

I see. That plugin might not delete ranges. @sgiehl can you confirm that?

@sgiehl commented on August 19th 2020 Member

It uses CoreAdminHome.invalidateArchivedReports without providing a period. Not sure if that method also invalidates ranges

@tsteur commented on August 19th 2020 Member

I just debugged and checked it should invalidate ranges as well since no period is set it would simply invalidate any period that matches the date range

query looked like this:

UPDATE matomo_archive_numeric_2020_01 SET value = 4 WHERE name LIKE 'done%'
                   AND idsite IN (1, 2, 3, ...)
                   AND (((date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) ...

@ibril15 any chance you can try to invalidate and archive the last 2 months again? and only select the range once archiving is fully done?

Generally I think there might be though an issue that ranges might not be archived again after a certain time or invalidated correctly maybe but I haven't really tried to reproduce this one in particular (eg fetch range, track more data in today, fetch range after the archive TTL again and check it's updated. Same test for tracking data in the past say 2 weeks ago or 2 months ago and see if ranges will be re-archived again)

@ibril15 commented on August 20th 2020

I used Invalidate Reports plugin to invalidate the archive again for the last month (Last 2 Months was not an option). Here's the output I got:

image

I gave it a few hours, and the results are the same.

@tsteur commented on August 20th 2020 Member

@ibril15 is there any chance you could create a super user login for us and send us the details to hello at matomo.org ? It might help understanding what's happening there.

@ibril15 commented on August 20th 2020

I'm not sure I'd be able to do that , but perhaps we can do a live screenshare of some sort. Or if there are specific data-points or logs I can provide, please let me know.

@tsteur commented on August 21st 2020 Member

@ibril15 unfortunately we don't do live screenshare for community support as it's quite time consuming and often not that easy to arrange things re timezone and makes things often take longer.

Could you otherwise check if you see this pattern also for other date ranges? I assume the archiving has definitely run since invalidating the archives?

@ibril15 commented on August 21st 2020

Yeah, the archiving should be done. I checked today again and it's the same issue.

2020-06-29 to 2020-07-05 also has the same problem. I checked and other Measurables (websites) also have the same problem. On other meaurables that have a longer history, I can see the same issue for dates like 2020-04-27 to 2020-05-03. But I don't see it for any date ranges in 2019. The first instance I see is from 2019-12-30 to 2020-01-05.

We have "Archive reports when viewed from the browser" set to "No", by the way, and we have a Cron job archiving reports.

@tsteur commented on August 25th 2020 Member

I can't reproduce this so far. Any chance you can update to the latest Matomo version and see if this is still happening?

And could you also go to Matomo Admin -> System Check and check what it says for the row Archive Cron?

@Littlericket commented on September 10th 2020

@tsteur don't know if this fits here as well, but we're seeing this behaviour for the report "search words without results". When viewing a month, it has more tabs than viewing the same for the current year. The result rows are less pages when viewing the yearly reports, compared to the month view:

aug 2020:
image

year 2020:
image

We indeed have visits from aug 2020 in the yearly date range, so calculation has been done for the year so far. Matomo is at 3.14.0.

@ibril15 commented on September 10th 2020

Sorry @tsteur, I missed the question. It says: " Managing processes via CLI: Ok "

Powered by GitHub Issue Mirror