Note: sounds like sum issue with date range invalidation.

@ibril15 do you mind letting us know how you invalidated the archive? Per API call or CLI command? Do you remember which parameters were used?

I used the Invalidate Reports plugin (v0.1.2). Selected All Segments, All Historical Data, and just the website with the issue.

I see. That plugin might not delete ranges. @sgiehl can you confirm that?

It uses CoreAdminHome.invalidateArchivedReports without providing a period. Not sure if that method also invalidates ranges

I just debugged and checked it should invalidate ranges as well since no period is set it would simply invalidate any period that matches the date range

query looked like this:

UPDATE matomo_archive_numeric_2020_01 SET value = 4 WHERE name LIKE 'done%'
                   AND idsite IN (1, 2, 3, ...)
                   AND (((date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) ...

@ibril15 any chance you can try to invalidate and archive the last 2 months again? and only select the range once archiving is fully done?

Generally I think there might be though an issue that ranges might not be archived again after a certain time or invalidated correctly maybe but I haven't really tried to reproduce this one in particular (eg fetch range, track more data in today, fetch range after the archive TTL again and check it's updated. Same test for tracking data in the past say 2 weeks ago or 2 months ago and see if ranges will be re-archived again)

I used Invalidate Reports plugin to invalidate the archive again for the last month (Last 2 Months was not an option). Here's the output I got:

I gave it a few hours, and the results are the same.

@ibril15 is there any chance you could create a super user login for us and send us the details to hello at matomo.org ? It might help understanding what's happening there.

I'm not sure I'd be able to do that , but perhaps we can do a live screenshare of some sort. Or if there are specific data-points or logs I can provide, please let me know.

@ibril15 unfortunately we don't do live screenshare for community support as it's quite time consuming and often not that easy to arrange things re timezone and makes things often take longer.

Could you otherwise check if you see this pattern also for other date ranges? I assume the archiving has definitely run since invalidating the archives?

Yeah, the archiving should be done. I checked today again and it's the same issue.

2020-06-29 to 2020-07-05 also has the same problem. I checked and other Measurables (websites) also have the same problem. On other meaurables that have a longer history, I can see the same issue for dates like 2020-04-27 to 2020-05-03. But I don't see it for any date ranges in 2019. The first instance I see is from 2019-12-30 to 2020-01-05.

We have "Archive reports when viewed from the browser" set to "No", by the way, and we have a Cron job archiving reports.

I can't reproduce this so far. Any chance you can update to the latest Matomo version and see if this is still happening?

And could you also go to Matomo Admin -> System Check and check what it says for the row Archive Cron?

@tsteur don't know if this fits here as well, but we're seeing this behaviour for the report "search words without results". When viewing a month, it has more tabs than viewing the same for the current year. The result rows are less pages when viewing the yearly reports, compared to the month view:

aug 2020:

year 2020:

We indeed have visits from aug 2020 in the yearly date range, so calculation has been done for the year so far. Matomo is at 3.14.0.

Sorry @tsteur, I missed the question. It says: " Managing processes via CLI: Ok "

@garrick-abine I'm following up with #16545 as it seems to me that it is the same issue.

@garrick-abine in Matomo Admin -> General Settings can you check if browser archiving is enabled?

And could you also go to Matomo Admin -> System Check and check what it says for the row Archive Cron? (similar to #16320 (comment) )

Invalidating the month of September might actually not invalidate all the different ranges. Do you remember how it was invalidated?

BTW in Matomo 4 we have made quite a few changes around this so the issue (which we can't reproduce so far) might be solved there already. Release will be mid-end November.

Browser archiving is disabled. We have a cron job setup for archiving (running every two hours) which runs on the reporter node. We have multiple tracker nodes.

For the invalidation, we used the Invalidate Reports plugin first. Then we tried the command line:
console -vv core:invalidate-report-data --dates=2020-09-01,2020-09-30 --sites=7

In system check we have:

Archive Cron: Managing processes via CLI: not supported (optional)
Last Successful Archiving Completion: The archiving process completed successfully 01:01:27 ago.

Is there anyone experiencing this issue where we could maybe get access to the database to look at the archive tables and partially log tables to verify the correct data to better understand what is happening there?

Sorry, we can't give direct DB access. We could do a screenshare if that would be helpful.

@tsteur Have you discovered any further information regarding this issue? Can I send some information from the DB that might help?

@garrick-abine sorry unfortunately I don't have any further information so far. We're planning to release Matomo 4 in 2-3 weeks and I think for now it might be easiest to simply give Matomo 4 a try and see if it fixes it since we changed quite a few things there. This week we have already released the first Matomo 4 RC version.

Thanks for contributing to this issue. As it has been a few months since the last activity and we believe this is likely not an issue anymore, we will now close this. If that's not the case, please do feel free to either reopen this issue or open a new one. We will gladly take a look again!