I seem to be having the same issue. Premised environment running 3.13.0. For me it seems the issue is when the week goes across months.
Date Range: Week From 2020-07-27 To 2020-08-02
If I select Custom Range for 2020-07-27 To 2020-08-01 (one day previous), I get 1,661 visits. If I add 2020-08-02 to the range, or use the pre-built Week option for that week, I get 1,407 visits. On 2020-08-02 I have 314 visits.
If I include 2020-08-02 in a range like 2020-07-28 to 2020-08-03 (doesn't include a whole week), the sum of the individual days equals the number of visits and everything looks right. If I change the range to 2020-07-26 to 2020-08-03 (includes a whole week and some extra days), the numbers are wrong again.
So it appears that the issue may be when calculating a preset range (like "Week"), maybe when the week crosses months? When I select a custom date range that includes a pre-set range like week, I'm assuming the range detects that and uses the pre-archived week. And if that pre-archived week is wrong, all the subsequent calculations that depend on it will also be wrong.
I tried to invalidate the archive and re-archive the data but that didn't help. Please let me know if I can do any further troubleshooting.
Thanks a lot.
Originally posted by @ibril15 in https://github.com/matomo-org/matomo/issues/15363#issuecomment-675508797
Note: sounds like sum issue with date range invalidation.
@ibril15 do you mind letting us know how you invalidated the archive? Per API call or CLI command? Do you remember which parameters were used?
I used the Invalidate Reports plugin (v0.1.2). Selected All Segments, All Historical Data, and just the website with the issue.
I see. That plugin might not delete ranges. @sgiehl can you confirm that?
CoreAdminHome.invalidateArchivedReports without providing a period. Not sure if that method also invalidates ranges
I just debugged and checked it should invalidate ranges as well since no period is set it would simply invalidate any period that matches the date range
query looked like this:
UPDATE matomo_archive_numeric_2020_01 SET value = 4 WHERE name LIKE 'done%' AND idsite IN (1, 2, 3, ...) AND (((date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) ...
@ibril15 any chance you can try to invalidate and archive the last 2 months again? and only select the range once archiving is fully done?
Generally I think there might be though an issue that ranges might not be archived again after a certain time or invalidated correctly maybe but I haven't really tried to reproduce this one in particular (eg fetch range, track more data in today, fetch range after the archive TTL again and check it's updated. Same test for tracking data in the past say 2 weeks ago or 2 months ago and see if ranges will be re-archived again)
I used Invalidate Reports plugin to invalidate the archive again for the last month (Last 2 Months was not an option). Here's the output I got:
I gave it a few hours, and the results are the same.
@ibril15 is there any chance you could create a super user login for us and send us the details to hello at matomo.org ? It might help understanding what's happening there.
I'm not sure I'd be able to do that , but perhaps we can do a live screenshare of some sort. Or if there are specific data-points or logs I can provide, please let me know.
@ibril15 unfortunately we don't do live screenshare for community support as it's quite time consuming and often not that easy to arrange things re timezone and makes things often take longer.
Could you otherwise check if you see this pattern also for other date ranges? I assume the archiving has definitely run since invalidating the archives?
Yeah, the archiving should be done. I checked today again and it's the same issue.
2020-06-29 to 2020-07-05 also has the same problem. I checked and other Measurables (websites) also have the same problem. On other meaurables that have a longer history, I can see the same issue for dates like 2020-04-27 to 2020-05-03. But I don't see it for any date ranges in 2019. The first instance I see is from 2019-12-30 to 2020-01-05.
We have "Archive reports when viewed from the browser" set to "No", by the way, and we have a Cron job archiving reports.
I can't reproduce this so far. Any chance you can update to the latest Matomo version and see if this is still happening?
And could you also go to
Matomo Admin -> System Check and check what it says for the row
@tsteur don't know if this fits here as well, but we're seeing this behaviour for the report "search words without results". When viewing a month, it has more tabs than viewing the same for the current year. The result rows are less pages when viewing the yearly reports, compared to the month view:
We indeed have visits from aug 2020 in the yearly date range, so calculation has been done for the year so far. Matomo is at 3.14.0.
Sorry @tsteur, I missed the question. It says: " Managing processes via CLI: Ok "
Matomo Admin -> General Settings can you check if browser archiving is enabled?
And could you also go to Matomo Admin -> System Check and check what it says for the row Archive Cron? (similar to https://github.com/matomo-org/matomo/issues/16320#issuecomment-690311288 )
Invalidating the month of September might actually not invalidate all the different ranges. Do you remember how it was invalidated?
BTW in Matomo 4 we have made quite a few changes around this so the issue (which we can't reproduce so far) might be solved there already. Release will be mid-end November.
Browser archiving is disabled. We have a cron job setup for archiving (running every two hours) which runs on the reporter node. We have multiple tracker nodes.
For the invalidation, we used the Invalidate Reports plugin first. Then we tried the command line:
console -vv core:invalidate-report-data --dates=2020-09-01,2020-09-30 --sites=7
In system check we have:
Archive Cron: Managing processes via CLI: not supported (optional)
Last Successful Archiving Completion: The archiving process completed successfully 01:01:27 ago.
Is there anyone experiencing this issue where we could maybe get access to the database to look at the archive tables and partially log tables to verify the correct data to better understand what is happening there?
Sorry, we can't give direct DB access. We could do a screenshare if that would be helpful.
@tsteur Have you discovered any further information regarding this issue? Can I send some information from the DB that might help?
@garrick-abine sorry unfortunately I don't have any further information so far. We're planning to release Matomo 4 in 2-3 weeks and I think for now it might be easiest to simply give Matomo 4 a try and see if it fixes it since we changed quite a few things there. This week we have already released the first Matomo 4 RC version.