@ibril15 opened this Issue on August 18th 2020

Hi All,

I seem to be having the same issue. Premised environment running 3.13.0. For me it seems the issue is when the week goes across months.

For Example:

Date Range: Week From 2020-07-27 To 2020-08-02

If I select Custom Range for 2020-07-27 To 2020-08-01 (one day previous), I get 1,661 visits. If I add 2020-08-02 to the range, or use the pre-built Week option for that week, I get 1,407 visits. On 2020-08-02 I have 314 visits.

If I include 2020-08-02 in a range like 2020-07-28 to 2020-08-03 (doesn't include a whole week), the sum of the individual days equals the number of visits and everything looks right. If I change the range to 2020-07-26 to 2020-08-03 (includes a whole week and some extra days), the numbers are wrong again.

So it appears that the issue may be when calculating a preset range (like "Week"), maybe when the week crosses months? When I select a custom date range that includes a pre-set range like week, I'm assuming the range detects that and uses the pre-archived week. And if that pre-archived week is wrong, all the subsequent calculations that depend on it will also be wrong.

I tried to invalidate the archive and re-archive the data but that didn't help. Please let me know if I can do any further troubleshooting.

Thanks a lot.


Originally posted by @ibril15 in https://github.com/matomo-org/matomo/issues/15363#issuecomment-675508797

@tsteur commented on August 18th 2020 Member

Note: sounds like sum issue with date range invalidation.

@tsteur commented on August 18th 2020 Member

@ibril15 do you mind letting us know how you invalidated the archive? Per API call or CLI command? Do you remember which parameters were used?

@ibril15 commented on August 18th 2020

I used the Invalidate Reports plugin (v0.1.2). Selected All Segments, All Historical Data, and just the website with the issue.

@tsteur commented on August 19th 2020 Member

I see. That plugin might not delete ranges. @sgiehl can you confirm that?

@sgiehl commented on August 19th 2020 Member

It uses CoreAdminHome.invalidateArchivedReports without providing a period. Not sure if that method also invalidates ranges

@tsteur commented on August 19th 2020 Member

I just debugged and checked it should invalidate ranges as well since no period is set it would simply invalidate any period that matches the date range

query looked like this:

UPDATE matomo_archive_numeric_2020_01 SET value = 4 WHERE name LIKE 'done%'
                   AND idsite IN (1, 2, 3, ...)
                   AND (((date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) OR (date1 <= ? AND ? <= date2) ...

@ibril15 any chance you can try to invalidate and archive the last 2 months again? and only select the range once archiving is fully done?

Generally I think there might be though an issue that ranges might not be archived again after a certain time or invalidated correctly maybe but I haven't really tried to reproduce this one in particular (eg fetch range, track more data in today, fetch range after the archive TTL again and check it's updated. Same test for tracking data in the past say 2 weeks ago or 2 months ago and see if ranges will be re-archived again)

@ibril15 commented on August 20th 2020

I used Invalidate Reports plugin to invalidate the archive again for the last month (Last 2 Months was not an option). Here's the output I got:


I gave it a few hours, and the results are the same.

@tsteur commented on August 20th 2020 Member

@ibril15 is there any chance you could create a super user login for us and send us the details to hello at matomo.org ? It might help understanding what's happening there.

@ibril15 commented on August 20th 2020

I'm not sure I'd be able to do that , but perhaps we can do a live screenshare of some sort. Or if there are specific data-points or logs I can provide, please let me know.

@tsteur commented on August 21st 2020 Member

@ibril15 unfortunately we don't do live screenshare for community support as it's quite time consuming and often not that easy to arrange things re timezone and makes things often take longer.

Could you otherwise check if you see this pattern also for other date ranges? I assume the archiving has definitely run since invalidating the archives?

@ibril15 commented on August 21st 2020

Yeah, the archiving should be done. I checked today again and it's the same issue.

2020-06-29 to 2020-07-05 also has the same problem. I checked and other Measurables (websites) also have the same problem. On other meaurables that have a longer history, I can see the same issue for dates like 2020-04-27 to 2020-05-03. But I don't see it for any date ranges in 2019. The first instance I see is from 2019-12-30 to 2020-01-05.

We have "Archive reports when viewed from the browser" set to "No", by the way, and we have a Cron job archiving reports.

@tsteur commented on August 25th 2020 Member

I can't reproduce this so far. Any chance you can update to the latest Matomo version and see if this is still happening?

And could you also go to Matomo Admin -> System Check and check what it says for the row Archive Cron?

@Littlericket commented on September 10th 2020

@tsteur don't know if this fits here as well, but we're seeing this behaviour for the report "search words without results". When viewing a month, it has more tabs than viewing the same for the current year. The result rows are less pages when viewing the yearly reports, compared to the month view:

aug 2020:

year 2020:

We indeed have visits from aug 2020 in the yearly date range, so calculation has been done for the year so far. Matomo is at 3.14.0.

@ibril15 commented on September 10th 2020

Sorry @tsteur, I missed the question. It says: " Managing processes via CLI: Ok "

@tsteur commented on October 7th 2020 Member

@garrick-abine I'm following up with #16545 as it seems to me that it is the same issue.

@garrick-abine in Matomo Admin -> General Settings can you check if browser archiving is enabled?


And could you also go to Matomo Admin -> System Check and check what it says for the row Archive Cron? (similar to https://github.com/matomo-org/matomo/issues/16320#issuecomment-690311288 )

Invalidating the month of September might actually not invalidate all the different ranges. Do you remember how it was invalidated?

@tsteur commented on October 7th 2020 Member

BTW in Matomo 4 we have made quite a few changes around this so the issue (which we can't reproduce so far) might be solved there already. Release will be mid-end November.

@garrick-abine commented on October 8th 2020

Browser archiving is disabled. We have a cron job setup for archiving (running every two hours) which runs on the reporter node. We have multiple tracker nodes.

For the invalidation, we used the Invalidate Reports plugin first. Then we tried the command line:
console -vv core:invalidate-report-data --dates=2020-09-01,2020-09-30 --sites=7

In system check we have:

Archive Cron: Managing processes via CLI: not supported (optional)
Last Successful Archiving Completion: The archiving process completed successfully 01:01:27 ago.

@tsteur commented on October 8th 2020 Member

Is there anyone experiencing this issue where we could maybe get access to the database to look at the archive tables and partially log tables to verify the correct data to better understand what is happening there?

@garrick-abine commented on October 9th 2020

Sorry, we can't give direct DB access. We could do a screenshare if that would be helpful.

@garrick-abine commented on November 5th 2020

@tsteur Have you discovered any further information regarding this issue? Can I send some information from the DB that might help?

@tsteur commented on November 6th 2020 Member

@garrick-abine sorry unfortunately I don't have any further information so far. We're planning to release Matomo 4 in 2-3 weeks and I think for now it might be easiest to simply give Matomo 4 a try and see if it fixes it since we changed quite a few things there. This week we have already released the first Matomo 4 RC version.

@mattab commented on April 1st 2021 Member

Thanks for contributing to this issue. As it has been a few months since the last activity and we believe this is likely not an issue anymore, we will now close this. If that's not the case, please do feel free to either reopen this issue or open a new one. We will gladly take a look again!

This Issue was closed on April 1st 2021
Powered by GitHub Issue Mirror