@nelhefni opened this Issue on April 8th 2021

Hello,

After updating an existing Goal, the next cron job will try to generate past reports.

My current settings:
rearchive_reports_in_past_last_n_months = last1

The cron job will then take up to 30 hours to complete!
Each weekly report took around 10 minutes. But it seems that the monthly report for February took 27 hours.
Thousands of partial reports will be inserted in the DB during the process

MariaDB [matomo]> select count(*) from matomo_archive_numeric_2021_03 where date1 = '2021-03-22' and period = 1 and name like '%Goal%';
+----------+
| count(*) |
+----------+
|   125936 |

We have 125936 Goal reports for a single day.
Filtering on the row including done and Goal:

MariaDB [matomo]> select count(*) from matomo_archive_numeric_2021_03 where date1 = '2021-03-22' and period = 1 and name like 'done%Goal%';
+----------+
| count(*) |
+----------+
|     3572 |
+----------+

Here is a small extract of the actual rows:

...
|    260168 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:28:35 |     5 |
|    260169 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:28:35 |     5 |
|    260194 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:16 |     5 |
|    260196 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:17 |     5 |
|    260197 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:17 |     5 |
|    260222 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:58 |     5 |
|    260224 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:59 |     5 |
|    260225 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:00 |     5 |
|    260250 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:40 |     5 |
|    260252 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:42 |     5 |
|    260253 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:42 |     5 |
|    260278 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:31:23 |     5 |
|    260280 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:31:24 |     5 |
|    260281 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:31:25 |     5 |
|    260306 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:05 |     5 |
|    260308 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:07 |     5 |
|    260309 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:08 |     5 |
|    260334 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:49 |     5 |
|    260336 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:50 |     5 |
|    260337 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:51 |     5 |
|    260362 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:33:32 |     5 |
|    260364 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:33:33 |     5 |
|    260365 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:33:34 |     5 |
|    260390 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:34:15 |     5 |
|    260392 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:34:16 |     5 |
|    260393 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:34:17 |     5 |
...

Goal plugin seems to be archiving segments for the same dates over and over.
Could you please have a look?

Your Environment

Matomo 4.2.1

@sgiehl commented on April 15th 2021 Member

I actually couldn't find any code that invalidates old archives when a Goal is updated.
@nelhefni are you using the Funnels plugin? That one might invalidate it's own reports when a Goal changes. But imho it shouldn't take so long to rearchive the Funnel reports. ping @diosmosis

@diosmosis commented on April 15th 2021 Member

@sgiehl this should be fixed by the PRs linked here: https://github.com/matomo-org/matomo/issues/17428#issuecomment-816429185 . We're waiting for verification there (otherwise I guess we'll close it before release).

@diosmosis commented on June 4th 2021 Member

Should be fixed by the PRs in 4.3, closing this one.

This Issue was closed on June 4th 2021
Powered by GitHub Issue Mirror