Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Goal report take ages to generate after an update #17434

Closed
nelhefni opened this issue Apr 8, 2021 · 3 comments
Closed

Goal report take ages to generate after an update #17434

nelhefni opened this issue Apr 8, 2021 · 3 comments
Labels
answered For when a question was asked and we referred to forum or answered it.

Comments

@nelhefni
Copy link

nelhefni commented Apr 8, 2021 •
edited

Hello,

After updating an existing Goal, the next cron job will try to generate past reports.

My current settings:
rearchive_reports_in_past_last_n_months = last1

The cron job will then take up to 30 hours to complete!
Each weekly report took around 10 minutes. But it seems that the monthly report for February took 27 hours.
Thousands of partial reports will be inserted in the DB during the process

MariaDB [matomo]> select count(*) from matomo_archive_numeric_2021_03 where date1 = '2021-03-22' and period = 1 and name like '%Goal%';
+----------+
| count(*) |
+----------+
|   125936 |

We have 125936 Goal reports for a single day.
Filtering on the row including done and Goal:

MariaDB [matomo]> select count(*) from matomo_archive_numeric_2021_03 where date1 = '2021-03-22' and period = 1 and name like 'done%Goal%';
+----------+
| count(*) |
+----------+
|     3572 |
+----------+

Here is a small extract of the actual rows:

...
|    260168 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:28:35 |     5 |
|    260169 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:28:35 |     5 |
|    260194 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:16 |     5 |
|    260196 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:17 |     5 |
|    260197 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:17 |     5 |
|    260222 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:58 |     5 |
|    260224 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:29:59 |     5 |
|    260225 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:00 |     5 |
|    260250 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:40 |     5 |
|    260252 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:42 |     5 |
|    260253 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:30:42 |     5 |
|    260278 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:31:23 |     5 |
|    260280 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:31:24 |     5 |
|    260281 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:31:25 |     5 |
|    260306 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:05 |     5 |
|    260308 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:07 |     5 |
|    260309 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:08 |     5 |
|    260334 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:49 |     5 |
|    260336 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:50 |     5 |
|    260337 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:32:51 |     5 |
|    260362 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:33:32 |     5 |
|    260364 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:33:33 |     5 |
|    260365 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:33:34 |     5 |
|    260390 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:34:15 |     5 |
|    260392 | done5aa0e3c4a2c285b80bcc0cd6e8a62f3e.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:34:16 |     5 |
|    260393 | donefea44bece172bc9696ae57c26888bf8a.Goals |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 20:34:17 |     5 |
...

Goal plugin seems to be archiving segments for the same dates over and over.
Could you please have a look?

Your Environment

Matomo 4.2.1
@nelhefni nelhefni added the Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. label Apr 8, 2021
@nelhefni nelhefni mentioned this issue Apr 8, 2021
Closed
@sgiehl
Copy link
Member

sgiehl commented Apr 15, 2021

I actually couldn't find any code that invalidates old archives when a Goal is updated.
@nelhefni are you using the Funnels plugin? That one might invalidate it's own reports when a Goal changes. But imho it shouldn't take so long to rearchive the Funnel reports. ping @diosmosis

@diosmosis
Copy link
Member

@sgiehl this should be fixed by the PRs linked here: #17428 (comment) . We're waiting for verification there (otherwise I guess we'll close it before release).

@diosmosis
Copy link
Member

Should be fixed by the PRs in 4.3, closing this one.

@mattab mattab added the answered For when a question was asked and we referred to forum or answered it. label Jul 27, 2021
@tsteur tsteur removed the Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. label Sep 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
answered For when a question was asked and we referred to forum or answered it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@diosmosis @tsteur @mattab @sgiehl @nelhefni