Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change rearchive end date from yesterday to today so it processes the last day correctly #18344

Closed
Starker3 opened this issue Nov 18, 2021 · 3 comments · Fixed by #18523
Closed
Assignees
Labels
Bug For errors / faults / flaws / inconsistencies etc.
Milestone

Comments

@Starker3
Copy link
Contributor

Refs DEV-2422 and potentially #18343

When a plugin that uses the rearchive_reports_in_past_last_n_months config setting and the setting is non-zero (Enabled) it seems as though the archiver skips processing the report for "Yesterday"

This was initially reported by a customer.

The easiest way to test this is with a Custom Report. As we can see in the screenshot below, this custom report was created on the 11th of November and according to the screenshot the 10th of November has no report data.
image

I tested this in my own instance and created a new Custom Report, the date of my server was the 14th of November and as we can see from the attached logs, the report for the 13th of November (Yesterday at the time of creation) was not processed for the Custom Report:
core-archive-custom-reports-yesterday.log

This Custom report has the following creation date:

select created_date from matomo_custom_reports where idcustomreport = 30;
+---------------------+
|created_date|
+---------------------+
|2021-11-14 22:31:21 |

It seems that it’s maybe skipping “Yesterday” because it would be processed during the normal core:archive run. But this might not always happen if the reports were already processed that day - in which case the “Yesterday” reports were maybe already processed and would maybe not be processed again.

@Starker3 Starker3 added the Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. label Nov 18, 2021
@tsteur tsteur added Bug For errors / faults / flaws / inconsistencies etc. and removed Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. labels Nov 18, 2021
@tsteur
Copy link
Member

tsteur commented Nov 18, 2021

Possible that changing Date::yesterday() to Date::today() helps in https://github.com/matomo-org/matomo/blob/4.6.0-rc1/core/Archive/ArchiveInvalidator.php#L460

Potentially the user or you @Starker3 could even try changing this and see if that works?

@tsteur tsteur added this to the 4.8.0 milestone Nov 18, 2021
@Starker3
Copy link
Contributor Author

@tsteur I quickly tested and it looks like it would work just changing that:

INFO [2021-11-18 01:02:11] 2930698    Will invalidate archived reports for today in site ID = 1's timezone (2021-11-18 00:00:00).
INFO [2021-11-18 01:02:11] 2930698    Will invalidate archived reports for yesterday in site ID = 1's timezone (2021-11-17 00:00:00).
INFO [2021-11-18 01:02:11] 2930698  Archived website id 1, period = day, date = 2021-11-18, segment = '', 0 visits found. Time elapsed: 0.518s
INFO [2021-11-18 01:02:11] 2930698  Archived website id 1, period = day, date = 2021-11-17, segment = '', plugin = CustomReports, report = 32, 0 visits found. Time elapsed: 0.518s
INFO [2021-11-18 01:02:11] 2930698  Archived website id 1, period = day, date = 2021-11-16, segment = '', plugin = CustomReports, report = 32, 4 visits found. Time elapsed: 0.518s

@tsteur
Copy link
Member

tsteur commented Nov 18, 2021

Great, thanks for confirming. I would create a PR for this but it also requires to make the tests work again so it's not done in 2 minutes.

@tsteur tsteur modified the milestones: 4.8.0, 4.7.0 Nov 18, 2021
@bx80 bx80 self-assigned this Dec 19, 2021
@justinvelluppillai justinvelluppillai changed the title Historical data for "Yesterday" may not always be processed Change rearchive end date from yesterday to today so it processes the last day correctly Feb 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug For errors / faults / flaws / inconsistencies etc.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants