@mattab opened this Issue on January 14th 2020 Member

Below is an example of two core:archive runs, one after another. The first one doesn't process the idsite = 1 because it Will ignore websites and help finish a previous started queue instead. IDs: 2.

Then the second archiving says Archiving was last executed without error 46s ago and no new tracking data for website id 1 since 2020-01-14 23:09:09 UTC (since the last successful archiving) although the previous successful archiving did not archive idsite=1. Instead we expected that it would look for "new tracking data" since the time where idsite=1 was actually last archived.

Proposal:

  • consider the time of "previous successful archiving for this specific idsite" when looking if there were new tracking data since that time.
INFO [2020-01-14 23:08:12] 16707  ---------------------------
INFO [2020-01-14 23:08:12] 16707  INIT
INFO [2020-01-14 23:08:12] 16707  Running Matomo 3.13.1-rc1 as Super User
INFO [2020-01-14 23:08:12] 16707  ---------------------------
INFO [2020-01-14 23:08:12] 16707  NOTES
INFO [2020-01-14 23:08:12] 16707  - Async process archiving supported, using CliMulti.
INFO [2020-01-14 23:08:12] 16707  - Reports for today will be processed at most every 900 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO [2020-01-14 23:08:12] 16707  - Reports for the current week/month/year will be requested at most every 3600 seconds.
INFO [2020-01-14 23:08:12] 16707  - Archiving was last executed without error 57 days 22 hours ago
INFO [2020-01-14 23:08:12] 16707  - Will process 3 other websites because the last time they were archived was on a different day (in the website's timezone) , IDs: 1, 2, 3
INFO [2020-01-14 23:08:12] 16707  - Will process 1 other websites because some old data reports have been invalidated (eg. using the Log Import script or the InvalidateReports plugin) , IDs: 3
INFO [2020-01-14 23:08:12] 16707  Will ignore websites and help finish a previous started queue instead. IDs: 2
INFO [2020-01-14 23:08:12] 16707  ---------------------------
INFO [2020-01-14 23:08:12] 16707  START
INFO [2020-01-14 23:08:12] 16707  Starting Matomo reports archiving...
INFO [2020-01-14 23:08:12] 16707  Day has finished for website id 2 since last run
INFO [2020-01-14 23:08:13] 16707  Will pre-process for website id = 2, period = day, date = last52
INFO [2020-01-14 23:08:13] 16707  - pre-processing all visits
INFO [2020-01-14 23:08:24] 16707  Archived website id = 2, period = day, 0 segments, 0 visits in last 52 days, 0 visits today, Time elapsed: 11.322s
INFO [2020-01-14 23:08:24] 16707  Will pre-process for website id = 2, period = week, date = last59
INFO [2020-01-14 23:08:24] 16707  - pre-processing all visits
INFO [2020-01-14 23:08:51] 16707  Archived website id = 2, period = week, 0 segments, 0 visits in last 59 weeks, 0 visits this week, Time elapsed: 27.592s
INFO [2020-01-14 23:08:54] 16707  Will pre-process for website id = 2, period = month, date = last52
INFO [2020-01-14 23:08:54] 16707  - pre-processing all visits
INFO [2020-01-14 23:08:59] 16707  Archived website id = 2, period = month, 0 segments, 0 visits in last 52 months, 0 visits this month, Time elapsed: 7.830s
INFO [2020-01-14 23:09:00] 16707  Will pre-process for website id = 2, period = year, date = last7
INFO [2020-01-14 23:09:00] 16707  - pre-processing all visits
INFO [2020-01-14 23:09:02] 16707  Archived website id = 2, period = year, 0 segments, 0 visits in last 7 years, 0 visits this year, Time elapsed: 2.731s
INFO [2020-01-14 23:09:02] 16707  Archived website id = 2, 4 API requests, Time elapsed: 49.511s [1/1 done]
INFO [2020-01-14 23:09:02] 16707  Done archiving!
INFO [2020-01-14 23:09:02] 16707  ---------------------------
INFO [2020-01-14 23:09:02] 16707  SUMMARY
INFO [2020-01-14 23:09:02] 16707  Total visits for today across archived websites: 0
INFO [2020-01-14 23:09:02] 16707  Archived today's reports for 1 websites
INFO [2020-01-14 23:09:02] 16707  Archived week/month/year for 1 websites
INFO [2020-01-14 23:09:02] 16707  Skipped 2 websites
INFO [2020-01-14 23:09:02] 16707  - 0 skipped because no new visit since the last script execution
INFO [2020-01-14 23:09:02] 16707  - 0 skipped because existing daily reports are less than 900 seconds old
INFO [2020-01-14 23:09:02] 16707  - 0 skipped because existing week/month/year periods reports are less than 3600 seconds old
INFO [2020-01-14 23:09:02] 16707  Total API requests: 4
INFO [2020-01-14 23:09:02] 16707  done: 1/1 100%, 0 vtoday, 1 wtoday, 1 wperiods, 4 req, 49731 ms, no error
INFO [2020-01-14 23:09:02] 16707  Time elapsed: 49.731s
INFO [2020-01-14 23:09:02] 16707  ---------------------------
INFO [2020-01-14 23:09:02] 16707  SCHEDULED TASKS
INFO [2020-01-14 23:09:02] 16707  Starting Scheduled tasks... 

[....]

$ ./console core:archive
INFO [2020-01-14 23:09:55] 18931  ---------------------------
INFO [2020-01-14 23:09:55] 18931  INIT
INFO [2020-01-14 23:09:55] 18931  Running Matomo 3.13.1-b1 as Super User
INFO [2020-01-14 23:09:55] 18931  ---------------------------
INFO [2020-01-14 23:09:55] 18931  NOTES
INFO [2020-01-14 23:09:55] 18931  - Async process archiving supported, using CliMulti.
INFO [2020-01-14 23:09:55] 18931  - Reports for today will be processed at most every 900 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO [2020-01-14 23:09:55] 18931  - Reports for the current week/month/year will be requested at most every 3600 seconds.
INFO [2020-01-14 23:09:55] 18931  - Archiving was last executed without error 46s ago
INFO [2020-01-14 23:09:55] 18931  - Will process 1 other websites because some old data reports have been invalidated (eg. using the Log Import script or the InvalidateReports plugin) , IDs: 3
INFO [2020-01-14 23:09:55] 18931  ---------------------------
INFO [2020-01-14 23:09:55] 18931  START
INFO [2020-01-14 23:09:55] 18931  Starting Matomo reports archiving...
INFO [2020-01-14 23:09:55] 18931  - The following websites do not use the tracker: 3
INFO [2020-01-14 23:09:55] 18931  - no new tracking data for website id 1 since 2020-01-14 23:09:09 UTC (since the last successful archiving)
INFO [2020-01-14 23:09:55] 18931  Skipped website id 1 as archiving is not needed
INFO [2020-01-14 23:09:55] 18931  - no new tracking data for website id 2 since 2020-01-14 23:09:09 UTC (since the last successful archiving)
INFO [2020-01-14 23:09:55] 18931  Skipped website id 2 as archiving is not needed
INFO [2020-01-14 23:09:55] 18931  Old report was invalidated for website id 3
INFO [2020-01-14 23:09:55] 18931  Will pre-process for website id = 3, period = day, date = last52
INFO [2020-01-14 23:09:55] 18931  - pre-processing all visits
INFO [2020-01-14 23:10:09] 18931  Archived website id = 3, period = day, 0 segments, 4 visits in last 52 days, 1 visits today, Time elapsed: 13.875s
INFO [2020-01-14 23:10:09] 18931  Will pre-process for website id = 3, period = week, date = last206
INFO [2020-01-14 23:10:09] 18931  - pre-processing all visits
INFO [2020-01-14 23:11:22] 18931  Archived website id = 3, period = week, 0 segments, 16 visits in last 206 weeks, 1 visits this week, Time elapsed: 72.509s
INFO [2020-01-14 23:11:22] 18931  Will pre-process for website id = 3, period = month, date = last52
INFO [2020-01-14 23:11:22] 18931  - pre-processing all visits
@tsteur commented on January 14th 2020 Member

Duplicate of https://github.com/matomo-org/matomo/issues/15127 ?

Even using a time of a specific idsite might not work reliably since some segments might be processed but not others.

Powered by GitHub Issue Mirror