Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The core:invalidate-report-data command is giving an invalid date error when given multiple dates #17146

Closed
nciske opened this issue Jan 25, 2021 · 6 comments
Labels
answered For when a question was asked and we referred to forum or answered it.

Comments

@nciske
Copy link

nciske commented Jan 25, 2021

I'm trying to invalidate the last week of archives so they can be re-run after some bulk updates.

I can get a single date to run just fine, but specifying multiple dates (per this FAQ) barfs out an invalid date error every time.

  • Matomo 4.1.1
  • PHP 7.4 on Ubuntu 20.x

Example from FAQ:
./console core:invalidate-report-data --dates=2012-01-01,2011-10-15 --sites=1,3,5

Works fine:
./console core:invalidate-report-data --sites=1,2 --dates=2021-01-25

Does not work fine:
./console core:invalidate-report-data --sites=1,2 --dates=2021-01-25,2021-01-24

Results:

Invalidating day periods in 2021-01-25,2021-01-24 [segment = ]...
Invalidating week periods in 2021-01-25,2021-01-24 [segment = ]...
Invalidating month periods in 2021-01-25,2021-01-24 [segment = ]...
Invalidating year periods in 2021-01-25,2021-01-24 [segment = ]...

ERROR [2021-01-25 21:11:36] 157296  Uncaught exception: /{matomo path}/public/core/Period/Range.php(155): Specified date range is invalid.

Specified date range is invalid.
  [Exception]
  Specified date range is invalid.
@nciske
Copy link
Author

nciske commented Jan 25, 2021

I've worked around this by putting each date in it's own line of a bash script, but I assume that is less performant than passing multiple dates in one go? Or not?

@tsteur
Copy link
Member

tsteur commented Jan 26, 2021

does it work to flip the date @nciske ?

./console core:invalidate-report-data --sites=1,2 --dates=2021-01-24,2021-01-25

@nciske
Copy link
Author

nciske commented Jan 27, 2021

It does work. Fascinating!

@tsteur
Copy link
Member

tsteur commented Jan 27, 2021

👍 great, thanks for confirming. I'll adjust the example in the FAQ as it was wrong there. Thanks @nciske

@tsteur tsteur closed this as completed Jan 27, 2021
@tsteur tsteur added the answered For when a question was asked and we referred to forum or answered it. label Jan 27, 2021
@jorgeuos
Copy link

Hi,

I noticed that the --help flag shows that:

--dates                      List of dates or date ranges to invalidate report data for, eg, 2015-01-03 or 2015-01-05,2015-02-12. (multiple values allowed)

But I get this error:

./console core:invalidate-report-data --dates=2021-11-29,2021-12-03,2021-12-08,2021-12-13,2021-12-14,2021-12-15,2021-12-16,2021-12-17,2021-12-20,2021-12-21,2021-12-24,2021-12-25,2021-12-27,2021-12-28,2021-12-29,2021-12-31 --sites=1 --periods=day --plugin=Funnels --dry-run
INFO [2022-01-22 21:07:53] 63288  Invalidating day periods in 2021-11-29,2021-12-03,2021-12-08,2021-12-13,2021-12-14,2021-12-15,2021-12-16,2021-12-17,2021-12-20,2021-12-21,2021-12-24,2021-12-25,2021-12-27,2021-12-28,2021-12-29,2021-12-31 [segment = ]...
ERROR [2022-01-22 21:07:53] 63288  Uncaught exception: /var/www/html/plugins/CoreAdminHome/Commands/InvalidateReportData.php(215): Invalid date or date range specifier '2021-11-29,2021-12-03,2021-12-08,2021-12-13,2021-12-14,2021-12-15,2021-12-16,2021-12-17,2021-12-20,2021-12-21,2021-12-24,2021-12-25,2021-12-27,2021-12-28,2021-12-29,2021-12-31' [Query: , CLI mode: 1]



  [InvalidArgumentException]
  Invalid date or date range specifier '2021-11-29,2021-12-03,2021-12-08,2021-12-13,2021-12-14,2021-12-15,2021-12-16,2021-12-17,2021-12-20,2021-12-21,2021-12-24,2021-12-25,2
  021-12-27,2021-12-28,2021-12-29,2021-12-31'






  [Exception]
  Date format must be: YYYY-MM-DD, or 'today' or 'yesterday' or any keyword supported by the strtotime function (see http://php.net/strtotime for more information): 2021-11-
  29,2021-12-03,2021-12-08,2021-12-13,2021-12-14,2021-12-15,2021-12-16,2021-12-17,2021-12-20,2021-12-21,2021-12-24,2021-12-25,2021-12-27,2021-12-28,2021-12-29,2021-12-31



core:invalidate-report-data [--dates="..."] [--sites="..."] [--periods="..."] [--segment="..."] [--cascade] [--dry-run] [--plugin="..."] [--ignore-log-deletion-limit]

I've tried without --dry-run too, but get the same error.
I guess single dates per command or a date-range, but not list of dates.
Or am I formatting the command wrong?

Full output of help:

./console core:invalidate-report-data --help
Usage:
 core:invalidate-report-data [--dates="..."] [--sites="..."] [--periods="..."] [--segment="..."] [--cascade] [--dry-run] [--plugin="..."] [--ignore-log-deletion-limit]

Options:
 --dates                      List of dates or date ranges to invalidate report data for, eg, 2015-01-03 or 2015-01-05,2015-02-12. (multiple values allowed)
 --sites                      List of site IDs to invalidate report data for, eg, "1,2,3,4" or "all" for all sites. (default: "all")
 --periods                    List of period types to invalidate report data for. Can be one or more of the following values: day, week, month, year or "all" for all of them. (default: "all")
 --segment                    List of segments to invalidate report data for. This can be the segment string itself, the segment name from the UI or the ID of the segment. If specifying the segment definition, make sure it is encoded properly (it should be the same as the segment parameter in the URL. (multiple values allowed)
 --cascade                    If supplied, invalidation will cascade, invalidating child period types even if they aren't specified in --periods. For example, if --periods=week, --cascade will cause the days within those weeks to be invalidated as well. If --periods=month, then weeks and days will be invalidated. Note: if a period falls partly outside of a date range, then --cascade will also invalidate data for child periods outside the date range. For example, if --dates=2015-09-14,2015-09-15 & --periods=week, --cascade will also invalidate all days within 2015-09-13,2015-09-19, even those outside the date range.
 --dry-run                    For tests. Runs the command w/o actually invalidating anything.
 --plugin                     To invalidate data for a specific plugin only.
 --ignore-log-deletion-limit  Ignore the log purging limit when invalidating archives. If a date is older than the log purging threshold (which means there should be no log data for it), we normally skip invalidating it in order to prevent losing any report data. In some cases, however it is useful, if, for example, your site was imported from Google, and there is never any log data.
 --help (-h)                  Display this help message
 --quiet (-q)                 Do not output any message
 --verbose (-v|vv|vvv)        Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
 --version (-V)               Display this application version
 --ansi                       Force ANSI output
 --no-ansi                    Disable ANSI output
 --no-interaction (-n)        Do not ask any interactive question
 --matomo-domain              Matomo URL (protocol and domain) eg. "http://matomo.example.org"
 --xhprof                     Enable profiling with XHProf

Help:
 Invalidate archived report data by date range, site and period. Invalidated archive data will be re-archived during the next core:archive run. If your log data has changed for some reason, this command can be used to make sure reports are generated using the new, changed log data.

@lauxjpn
Copy link

lauxjpn commented Apr 4, 2023

The main misconception here seems to be, that users think that --dates=date1,date2 is a list of individual dates, when in fact, it is interpreted as a date range from date1 to date2 (inclusive), as can be verified by using the --verbose option:

php ./console core:invalidate-report-data --dates=2023-03-30,2023-04-02 --sites=1,2 --verbose

Invalidating day periods in 2023-03-30,2023-04-05 [segment = ]...
Success. The following dates were invalidated successfully: 2023-03-30, 2023-03-31, 2023-04-01, 2023-04-02

For multiple individual dates, use --dates=date1 --dates=date2 --dates=date3:

php ./console core:invalidate-report-data --dates=2023-03-30 --dates=2023-04-02 --sites=1,2 --verbose

Invalidating day periods in 2023-03-30 [segment = ]...
Success. The following dates were invalidated successfully: 2023-03-30
Invalidating day periods in 2023-04-02 [segment = ]...
Success. The following dates were invalidated successfully: 2023-04-02

The docs at How do I tell Matomo to invalidate the past historical reports so they can be re-processed from the logs? don't explain what the syntax in the sample code does, so it is not surprising that it is being misinterpreted by users.

It should be clarified by mentioning both, the --dates=date1,date2 and the --dates=date1 --dates=date2 --dates=date3 syntax.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
answered For when a question was asked and we referred to forum or answered it.
Projects
None yet
Development

No branches or pull requests

4 participants