Hello,
After updating an existing Goal, the next cron job will try to generate past reports.
My current settings:rearchive_reports_in_past_last_n_months = last1
The cron job will then take up to 30 hours to complete!
Each weekly report took around 10 minutes. But it seems that the monthly report for February took 27 hours.
Thousands of partial reports will be inserted in the DB during the process
MariaDB [matomo]> select count(*) from matomo_archive_numeric_2021_03 where date1 = '2021-03-22' and period = 1 and name like '%Goal%';
+----------+
| count(*) |
+----------+
| 125936 |
We have 125936 Goal reports for a single day.
Filtering on the row including done
and Goal
:
MariaDB [matomo]> select count(*) from matomo_archive_numeric_2021_03 where date1 = '2021-03-22' and period = 1 and name like 'done%Goal%';
+----------+
| count(*) |
+----------+
| 3572 |
+----------+
Here is a small extract of the actual rows:
...
| 260168 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:28:35 | 5 |
| 260169 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:28:35 | 5 |
| 260194 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:29:16 | 5 |
| 260196 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:29:17 | 5 |
| 260197 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:29:17 | 5 |
| 260222 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:29:58 | 5 |
| 260224 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:29:59 | 5 |
| 260225 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:30:00 | 5 |
| 260250 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:30:40 | 5 |
| 260252 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:30:42 | 5 |
| 260253 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:30:42 | 5 |
| 260278 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:31:23 | 5 |
| 260280 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:31:24 | 5 |
| 260281 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:31:25 | 5 |
| 260306 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:32:05 | 5 |
| 260308 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:32:07 | 5 |
| 260309 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:32:08 | 5 |
| 260334 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:32:49 | 5 |
| 260336 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:32:50 | 5 |
| 260337 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:32:51 | 5 |
| 260362 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:33:32 | 5 |
| 260364 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:33:33 | 5 |
| 260365 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:33:34 | 5 |
| 260390 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:34:15 | 5 |
| 260392 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:34:16 | 5 |
| 260393 | donefea44bece172bc9696ae57c26888bf8a.Goals | 1 | 2021-03-22 | 2021-03-22 | 1 | 2021-04-05 20:34:17 | 5 |
...
Goal plugin seems to be archiving segments for the same dates over and over.
Could you please have a look?
Matomo 4.2.1
I actually couldn't find any code that invalidates old archives when a Goal is updated.
@nelhefni are you using the Funnels plugin? That one might invalidate it's own reports when a Goal changes. But imho it shouldn't take so long to rearchive the Funnel reports. ping @diosmosis
@sgiehl this should be fixed by the PRs linked here: https://github.com/matomo-org/matomo/issues/17428#issuecomment-816429185 . We're waiting for verification there (otherwise I guess we'll close it before release).