Goal report take ages to generate after an update #17434

Hello,

After updating an existing Goal, the next cron job will try to generate past reports.

My current settings:
rearchive_reports_in_past_last_n_months = last1

The cron job will then take up to 30 hours to complete!
Each weekly report took around 10 minutes. But it seems that the monthly report for February took 27 hours.
Thousands of partial reports will be inserted in the DB during the process

MariaDB [matomo]> select count(*) from matomo_archive_numeric_2021_03 where date1 = '2021-03-22' and period = 1 and name like '%Goal%';
+----------+
| count(*) |
+----------+
|   125936 |

We have 125936 Goal reports for a single day.
Filtering on the row including done and Goal:

MariaDB [matomo]> select count(*) from matomo_archive_numeric_2021_03 where date1 = '2021-03-22' and period = 1 and name like 'done%Goal%';
+----------+
| count(*) |
+----------+
|     3572 |
+----------+

Here is a small extract of the actual rows:

...
|    260168 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:28:35 |     5 |
|    260169 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:28:35 |     5 |
|    260194 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:16 |     5 |
|    260196 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:17 |     5 |
|    260197 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:17 |     5 |
|    260222 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:58 |     5 |
|    260224 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:59 |     5 |
|    260225 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:00 |     5 |
|    260250 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:40 |     5 |
|    260252 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:42 |     5 |
|    260253 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:42 |     5 |
|    260278 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:31:23 |     5 |
|    260280 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:31:24 |     5 |
|    260281 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:31:25 |     5 |
|    260306 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:05 |     5 |
|    260308 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:07 |     5 |
|    260309 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:08 |     5 |
|    260334 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:49 |     5 |
|    260336 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:50 |     5 |
|    260337 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:51 |     5 |
|    260362 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:33:32 |     5 |
|    260364 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:33:33 |     5 |
|    260365 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:33:34 |     5 |
|    260390 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:34:15 |     5 |
|    260392 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:34:16 |     5 |
|    260393 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:34:17 |     5 |
...

Goal plugin seems to be archiving segments for the same dates over and over.
Could you please have a look?

Your Environment

Matomo 4.2.1

I actually couldn't find any code that invalidates old archives when a Goal is updated.
@nelhefni are you using the Funnels plugin? That one might invalidate it's own reports when a Goal changes. But imho it shouldn't take so long to rearchive the Funnel reports. ping @diosmosis

@sgiehl this should be fixed by the PRs linked here: https://github.com/matomo-org/matomo/issues/17428#issuecomment-816429185 . We're waiting for verification there (otherwise I guess we'll close it before release).

Should be fixed by the PRs in 4.3, closing this one.