Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression: some daily reports disappear from UI, and weekly reports are corrupted #17428

Closed
nelhefni opened this issue Apr 7, 2021 · 12 comments · May be fixed by nelhefni/matomo#2
Closed

Regression: some daily reports disappear from UI, and weekly reports are corrupted #17428

nelhefni opened this issue Apr 7, 2021 · 12 comments · May be fixed by nelhefni/matomo#2
Labels
answered For when a question was asked and we referred to forum or answered it.

Comments

@nelhefni
Copy link

nelhefni commented Apr 7, 2021 •
edited

Hello,

We think PR #17216 has introduced some regressions to the archiving process since Matomo 4.2.0

FYI, browser archiving is disabled:

enable_browser_archiving_triggering = 0
browser_archiving_disabled_enforce = 1
archiving_range_force_on_browser_request = 0

For a given plugin, we are facing 2 issues since the update to Matomo 4.2.0,

  • Issue#1: Some daily reports disappear from the UI, even though delete_reports_enable is set to 0
  • Issue#2: Some weekly reports have inaccurate data: gather the values of only some days of the week.

This seems to be related to the changes made with the handling of partial archives.
Indeed, we have a lot of partial archives in our DB, which doesn't contain the reports of our impacted plugins:

MariaDB [matomo]> select * from matomo_archive_numeric_2021_03 where idarchive = 46176;

| idarchive | name        | idsite   | date1      | date2      | period    | ts_archived         | value    |

|     46176 | myPlugin    |        1 | 2021-03-22 | 2021-03-22 |         1 | 2021-03-23 00:05:52 |       10 |
|     46176 | otherPlugin |        1 | 2021-03-22 | 2021-03-22 |         1 | 2021-03-23 00:05:52 |       20 |
|     46176 | done        |        1 | 2021-03-22 | 2021-03-22 |         1 | 2021-03-23 00:05:52 |        1 |

MariaDB [matomo]> select * from matomo_archive_numeric_2021_03 where idarchive = 56405;

| idarchive | name        | idsite   | date1      | date2      | period    | ts_archived         | value    |

|     56405 | otherPlugin |        1 | 2021-03-22 | 2021-03-22 |         1 | 2021-04-05 18:52:03 |       20 |
|     56405 | done        |        1 | 2021-03-22 | 2021-03-22 |         1 | 2021-04-05 18:52:03 |        5 |

As you can see, myPlugin report is not part of partial archive 56405 (partial because the value of done row is 5), but it is nevertheless selected since Matomo 4.2.0, as ts_archived is more recent.

Since #17216, checkAuthorizedToArchive = false is passed to Rules::getSelectableDoneFlagValues during the retrival of archive to display in the UI

matomo/core/DataAccess/ArchiveSelector.php

Line 370 in 2c30dc8

$possibleValues = Rules::getSelectableDoneFlagValues($includeInvalidated, null, $checkAuthorizedToArchive = false);

As a consequence, both ArchiveWriter::DONE_PARTIAL and ArchiveWriter::DONE_INVALIDATED are added to the list of accepted value. Which explains why archive 56405 is selected instead of 46176 in our case.

matomo/core/ArchiveProcessor/Rules.php

Line 321 in 2c30dc8

if (!$checkAuthorizedToArchive || !Rules::isRequestAuthorizedToArchive($params)) {

Thus, as soon as archive 56405 was created, myPlugin report disappeared from the UI.
I don't think ArchiveWriter::DONE_PARTIAL should be part of the list of accepted values.
We have been able to fix this issue#1 with this PR #17425

Concerning issue#2, we have been able to fix it by reverting the changes done in the same PR to ArchiveSelector::findArchiveDataWithLatestTsArchived, and then invalidating faulty reports and re-running the cron job.

Your Environment

Matomo 4.2.1
@nelhefni nelhefni added the Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. label Apr 7, 2021
This was referenced Apr 7, 2021
Closed
Open
@diosmosis
Copy link
Member

Hi @nelhefni, thanks for the detailed report. Partial archives are meant to be included in the entire list of all idarchives to look through. We don't select only one archive, but the latest DONE_OK or DONE_INVALIDATED archive, then all DONE_PARTIAL archives that are later. We then look for data in all of those archives. Using this knowledge, can you take another look? Or if possible can you provide access so we can take a closer look?

@nelhefni
Copy link
Author

nelhefni commented Apr 8, 2021 •
edited

Hi @diosmosis, thanks for the fast answer.
I've been tracing the code, and ArchiveSelector::getArchiveIds returns only one idarchive

matomo/core/DataAccess/ArchiveSelector.php

Line 198 in 2c30dc8

$archiveIds = $db->fetchAll($sql, $bind); 

archiveIds = 
Array
(
  [0] => Array
  (
    [idsite] => 1
    [date1] => 2021-03-22
    [date2] => 2021-03-22
    [archives] =>
    56405|done|5,53884|done|5,49890|done.Funnels|5,46176|done|1
  )
)

As there is only one element in $idarchives array, we loop only once in

matomo/core/DataAccess/ArchiveSelector.php

Line 202 in 2c30dc8

foreach ($archiveIds as $row) {

then,

matomo/core/DataAccess/ArchiveSelector.php

Line 206 in 2c30dc8

$pairs = explode(',', $archives); 

$pairs = 
Array
(
  [0] => 56405|done|5
  [1] => 53884|done|5
  [2] => 49890|done.Funnels|5
  [3] => 46176|done|1
)

then we break, because idarchive 56405 contains done without any dot

matomo/core/DataAccess/ArchiveSelector.php

Line 212 in 2c30dc8

break; // found the all plugins archive, don't need to look in older archives since we have everything here

finally, we have only one idarchive returned

matomo/core/Archive.php

Line 638 in 2c30dc8

$idarchivesByReport = ArchiveSelector::getArchiveIds( 

$idarchivesByReport =
Array
(
  [done] => Array
  (
    [2021-03-22,2021-03-22] => Array
    (
      [0] => 56405
    )
  )
)

@diosmosis
Copy link
Member

@nelhefni thanks again for the detailed post. That code looks broken but it's actually working correctly, the problem is that archives like 56405|done|5 should never exist... partial archives have to be specific to a plugin, they should never have a done flag of just done.

Can you see what reports are in 56405 (eg, what names are in the blob table for 56405)?

I think we can put in some code making sure partial archives are never saved w/ that done flag, and modify the code you reference just in case this edge case happens in some other way, but I can't think of how this might happen.

@nelhefni
Copy link
Author

nelhefni commented Apr 8, 2021 •
edited

@diosmosis thanks for the input, the issue could then be a side effect of #17434

Here are the reports for idarchive 56405:


MariaDB [matomo]> select * from matomo_archive_numeric_2021_03 where idarchive = 56405;

| idarchive | name                               | idsite | date1      | date2      | period | ts_archived         | value   |

|    56405 | Actions_nb_outlinks                |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Actions_nb_pageviews               |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Actions_nb_uniq_outlinks           |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Actions_nb_uniq_pageviews          |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | bounce_count                       |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | done                               |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |        5 |
|    56405 | Goal_10_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_10_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_10_revenue                    |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |     99.9 |
|    56405 | Goal_11_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_11_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_12_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_12_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_13_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_13_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_14_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_14_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_15_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_15_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_16_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_16_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_17_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_17_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_18_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_18_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_19_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_19_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_1_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_1_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_20_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_20_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_21_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_21_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_22_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_22_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_23_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_23_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_24_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_24_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_25_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_25_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_26_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_26_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_27_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_27_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_28_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_28_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_29_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_29_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_2_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_2_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_30_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_30_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_31_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_31_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_3_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_3_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_4_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_4_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_5_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_5_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_6_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_6_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_7_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_7_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_8_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_8_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_9_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_9_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_nb_conversions                |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_nb_visits_converted           |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_revenue                       |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |     99.9 |
|    56405 | max_actions                        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | nb_actions                         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | nb_uniq_visitors                   |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | nb_users                           |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | nb_visits                          |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | nb_visits_converted                |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | PagePerformance_domcompletion_hits |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_domcompletion_time |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_domprocessing_hits |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_domprocessing_time |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_network_hits       |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_network_time       |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_onload_hits        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_onload_time        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_pageload_hits      |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_pageload_time      |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_servery_time       |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_server_hits        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_transfer_hits      |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_transfer_time      |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Referrers_distinctCampaigns        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Referrers_distinctKeywords         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Referrers_distinctSearchEngines    |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Referrers_distinctSocialNetworks   |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Referrers_distinctWebsites         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Referrers_distinctWebsitesUrls     |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | sum_visit_length                   |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | UserCountry_distinctCountries      |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |        1 |

Which doesn't include myPlugin

@diosmosis
Copy link
Member

@nelhefni the Goals plugin doesn't have any logic to rearchive in the past when it changes: https://github.com/matomo-org/matomo/blob/4.x-dev/plugins/Goals/API.php#L198-L232. The funnels plugin does, which is connected, but then I would expect to see Funnels reports in this table. Can you try editing a goal or creating a goal, then checking what gets added to archive_invalidations (then truncate the table or take note of the max idinvalidation and delete everything greater than it to avoid launching any archiving)?

Based on the archive contents above it seems like that archive should have a done value of DONE_OK (or 1). I guess it might be possible that because it gets set as value = 5, it thinks the archiving never happens and keeps trying... I'll have a look at the code when I can to see where it might be possible to set the value as 5 for an archive w/ name = 'done'.

@nelhefni
Copy link
Author

nelhefni commented Apr 8, 2021 •
edited

@diosmosis I did update the funnel attached to one of my goals.
ID of the funnel changed from 32 to 33.
After the update, nothing was added to the archive_invalidations table, but 2 rows were removed:

| idinvalidation | idarchive | name                                                | idsite | date1      | date2      | period | ts_invalidated      | ts_started          | status | report |

|        1260359 |    457962 | done.Funnels                                        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-03-25 06:06:09 | 2021-03-26 20:04:28 |      0 | 32     |
|        1260644 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-03-25 06:06:49 | NULL                |      0 | 32     |

Then, when the cron job started, the invalidation process added a lot of new rows to the archive_invalidations table:

select * from matomo_archive_invalidations where name like '%unnel%' and report = 33;

| idinvalidation | idarchive | name                                                | idsite | date1      | date2      | period | ts_invalidated      | ts_started | status | report |

|        1429392 |     54863 | done.Funnels                                        |      1 | 2021-03-01 | 2021-03-01 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429393 |     54894 | done.Funnels                                        |      1 | 2021-03-01 | 2021-03-07 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429394 |    190726 | done.Funnels                                        |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429395 |     54862 | done.Funnels                                        |      1 | 2021-03-02 | 2021-03-02 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429396 |     54858 | done.Funnels                                        |      1 | 2021-03-03 | 2021-03-03 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429397 |     54827 | done.Funnels                                        |      1 | 2021-03-04 | 2021-03-04 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429398 |     54825 | done.Funnels                                        |      1 | 2021-03-05 | 2021-03-05 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429399 |     54822 | done.Funnels                                        |      1 | 2021-03-06 | 2021-03-06 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429400 |     54447 | done.Funnels                                        |      1 | 2021-03-07 | 2021-03-07 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429401 |     54446 | done.Funnels                                        |      1 | 2021-03-08 | 2021-03-08 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429402 |     54472 | done.Funnels                                        |      1 | 2021-03-08 | 2021-03-14 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429403 |     54443 | done.Funnels                                        |      1 | 2021-03-09 | 2021-03-09 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429404 |     54411 | done.Funnels                                        |      1 | 2021-03-10 | 2021-03-10 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429405 |     54410 | done.Funnels                                        |      1 | 2021-03-11 | 2021-03-11 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429406 |     54407 | done.Funnels                                        |      1 | 2021-03-12 | 2021-03-12 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429407 |     54019 | done.Funnels                                        |      1 | 2021-03-13 | 2021-03-13 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429408 |     54015 | done.Funnels                                        |      1 | 2021-03-14 | 2021-03-14 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429409 |     53983 | done.Funnels                                        |      1 | 2021-03-15 | 2021-03-15 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429410 |     54017 | done.Funnels                                        |      1 | 2021-03-15 | 2021-03-21 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429411 |     53982 | done.Funnels                                        |      1 | 2021-03-16 | 2021-03-16 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429412 |     53978 | done.Funnels                                        |      1 | 2021-03-17 | 2021-03-17 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429413 |     53947 | done.Funnels                                        |      1 | 2021-03-18 | 2021-03-18 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429414 |     53946 | done.Funnels                                        |      1 | 2021-03-19 | 2021-03-19 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429415 |     53942 | done.Funnels                                        |      1 | 2021-03-20 | 2021-03-20 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429416 |     53894 | done.Funnels                                        |      1 | 2021-03-21 | 2021-03-21 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429417 |     53890 | done.Funnels                                        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429418 |     67174 | done.Funnels                                        |      1 | 2021-03-22 | 2021-03-28 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429419 |     53889 | done.Funnels                                        |      1 | 2021-03-23 | 2021-03-23 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429420 |      NULL | done.Funnels                                        |      1 | 2021-03-24 | 2021-03-24 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429421 |      NULL | done.Funnels                                        |      1 | 2021-03-25 | 2021-03-25 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429422 |      NULL | done.Funnels                                        |      1 | 2021-03-26 | 2021-03-26 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429423 |      NULL | done.Funnels                                        |      1 | 2021-03-27 | 2021-03-27 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429424 |      NULL | done.Funnels                                        |      1 | 2021-03-28 | 2021-03-28 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429425 |      NULL | done.Funnels                                        |      1 | 2021-03-29 | 2021-03-29 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429426 |      NULL | done.Funnels                                        |      1 | 2021-03-29 | 2021-04-04 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429427 |      NULL | done.Funnels                                        |      1 | 2021-03-30 | 2021-03-30 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429428 |      NULL | done.Funnels                                        |      1 | 2021-03-31 | 2021-03-31 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429455 |    457962 | done.Funnels                                        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429456 |      NULL | done.Funnels                                        |      1 | 2021-04-01 | 2021-04-01 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429457 |      NULL | done.Funnels                                        |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429458 |      NULL | done.Funnels                                        |      1 | 2021-04-02 | 2021-04-02 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429459 |      NULL | done.Funnels                                        |      1 | 2021-04-03 | 2021-04-03 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429460 |      NULL | done.Funnels                                        |      1 | 2021-04-04 | 2021-04-04 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429461 |      NULL | done.Funnels                                        |      1 | 2021-04-05 | 2021-04-05 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429462 |      NULL | done.Funnels                                        |      1 | 2021-04-05 | 2021-04-11 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429463 |      NULL | done.Funnels                                        |      1 | 2021-04-06 | 2021-04-06 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429471 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-01 | 2021-03-01 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429472 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-01 | 2021-03-07 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429473 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429474 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-02 | 2021-03-02 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429475 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-03 | 2021-03-03 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429476 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-04 | 2021-03-04 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429477 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-05 | 2021-03-05 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429478 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-06 | 2021-03-06 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429479 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-07 | 2021-03-07 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429480 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-08 | 2021-03-08 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429481 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-08 | 2021-03-14 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429482 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-09 | 2021-03-09 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429483 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-10 | 2021-03-10 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429484 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-11 | 2021-03-11 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429485 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-12 | 2021-03-12 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429486 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-13 | 2021-03-13 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429487 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-14 | 2021-03-14 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429488 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-15 | 2021-03-15 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429489 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-15 | 2021-03-21 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429490 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-16 | 2021-03-16 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429491 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-17 | 2021-03-17 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429492 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-18 | 2021-03-18 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429493 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-19 | 2021-03-19 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429494 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-20 | 2021-03-20 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429495 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-21 | 2021-03-21 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429496 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429497 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-22 | 2021-03-28 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429498 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-23 | 2021-03-23 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429499 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-24 | 2021-03-24 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429500 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-25 | 2021-03-25 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429501 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-26 | 2021-03-26 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429502 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-27 | 2021-03-27 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429503 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-28 | 2021-03-28 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429504 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-29 | 2021-03-29 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429505 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-29 | 2021-04-04 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429506 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-30 | 2021-03-30 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429507 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-31 | 2021-03-31 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429534 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429535 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-01 | 2021-04-01 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429536 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429537 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-02 | 2021-04-02 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429538 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-03 | 2021-04-03 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429539 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-04 | 2021-04-04 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429540 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-05 | 2021-04-05 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429541 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-05 | 2021-04-11 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429542 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-06 | 2021-04-06 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429550 |     54888 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-01 | 2021-03-01 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429551 |     55095 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-01 | 2021-03-07 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429552 |     96258 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429553 |     54879 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-02 | 2021-03-02 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429554 |     54868 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-03 | 2021-03-03 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429555 |     54853 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-04 | 2021-03-04 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429556 |     54844 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-05 | 2021-03-05 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429557 |     54831 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-06 | 2021-03-06 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429558 |     54672 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-07 | 2021-03-07 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429559 |     54464 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-08 | 2021-03-08 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429560 |     54677 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-08 | 2021-03-14 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429561 |     54452 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-09 | 2021-03-09 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429562 |     54437 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-10 | 2021-03-10 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429563 |     54428 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-11 | 2021-03-11 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429564 |     54416 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-12 | 2021-03-12 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429565 |     54401 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-13 | 2021-03-13 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429566 |     54390 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-14 | 2021-03-14 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429567 |     54008 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-15 | 2021-03-15 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429568 |     54223 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-15 | 2021-03-21 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429569 |     54000 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-16 | 2021-03-16 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429570 |     53988 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-17 | 2021-03-17 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429571 |     53973 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-18 | 2021-03-18 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429572 |     53964 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-19 | 2021-03-19 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429573 |     53952 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-20 | 2021-03-20 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429574 |     53936 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-21 | 2021-03-21 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429575 |     53907 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429576 |     67373 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-22 | 2021-03-28 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429577 |     53898 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-23 | 2021-03-23 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429578 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-24 | 2021-03-24 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429579 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-25 | 2021-03-25 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429580 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-26 | 2021-03-26 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429581 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-27 | 2021-03-27 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429582 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-28 | 2021-03-28 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429583 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-29 | 2021-03-29 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429584 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-29 | 2021-04-04 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429585 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-30 | 2021-03-30 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429586 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-31 | 2021-03-31 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429613 |    891194 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429614 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-01 | 2021-04-01 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429615 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429616 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-02 | 2021-04-02 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429617 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-03 | 2021-04-03 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429618 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-04 | 2021-04-04 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429619 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-05 | 2021-04-05 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429620 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-05 | 2021-04-11 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429621 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-06 | 2021-04-06 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429629 |     54887 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-01 | 2021-03-01 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429630 |     55096 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-01 | 2021-03-07 |      2 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429631 |     96257 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429632 |     54878 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-02 | 2021-03-02 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429633 |     54867 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-03 | 2021-03-03 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429634 |     54852 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-04 | 2021-03-04 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429635 |     54843 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-05 | 2021-03-05 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429636 |     54832 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-06 | 2021-03-06 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429637 |     54475 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-07 | 2021-03-07 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429638 |     54463 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-08 | 2021-03-08 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429639 |     54678 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-08 | 2021-03-14 |      2 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429640 |     54451 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-09 | 2021-03-09 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429641 |     54436 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-10 | 2021-03-10 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429642 |     54427 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-11 | 2021-03-11 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429643 |     54415 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-12 | 2021-03-12 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429644 |     54400 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-13 | 2021-03-13 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429645 |     54389 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-14 | 2021-03-14 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429646 |     54009 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-15 | 2021-03-15 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429647 |     54222 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-15 | 2021-03-21 |      2 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429648 |     53999 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-16 | 2021-03-16 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429649 |     53987 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-17 | 2021-03-17 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429650 |     53972 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-18 | 2021-03-18 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429651 |     53963 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-19 | 2021-03-19 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429652 |     53951 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-20 | 2021-03-20 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429653 |     53937 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-21 | 2021-03-21 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429654 |     53908 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429655 |     67372 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-22 | 2021-03-28 |      2 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429656 |     53899 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-23 | 2021-03-23 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429657 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-24 | 2021-03-24 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429658 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-25 | 2021-03-25 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429659 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-26 | 2021-03-26 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429660 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-27 | 2021-03-27 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429661 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-28 | 2021-03-28 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429662 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-29 | 2021-03-29 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429663 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-29 | 2021-04-04 |      2 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429664 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-30 | 2021-03-30 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429665 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-31 | 2021-03-31 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429692 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429693 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-01 | 2021-04-01 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429694 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429695 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-02 | 2021-04-02 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429696 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-03 | 2021-04-03 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429697 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-04 | 2021-04-04 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429698 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-05 | 2021-04-05 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429699 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-05 | 2021-04-11 |      2 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429700 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-06 | 2021-04-06 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |

This seems to make sense, as we have 3 segments, and go back to March 1st because of the following setting:
rearchive_reports_in_past_last_n_months = last1
Not too sure why there are a lot of idarchive set to NULL though.

Moreover, If I look for the corresponding idarchive in matomo_archive_numeric_2021_**, I find only one row:

select * from matomo_archive_numeric_2021_03 where idarchive = 54863;

| idarchive | name         | idsite | date1      | date2      | period | ts_archived         | value |

|     54863 | done.Funnels |      1 | 2021-03-01 | 2021-03-01 |      1 | 2021-03-25 06:35:35 |     4 |

I'm not sure to understand the purpose of this done.Funnels, could you please explain the concept behind it?

Also, I deleted manually all the rows for period 3, as I want to avoid the issue described here: #17434,

delete from matomo_archive_invalidations where name like '%unnel%' and period = 3 and report = 33;

| idinvalidation | idarchive | name                                         | idsite | date1      | date2      | period | ts_invalidated      | ts_started | status | report |

|        1429394 |    190726 | done.Funnels                                 |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429457 |      NULL | done.Funnels                                 |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429473 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429536 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429552 |     96258 | done0454a4912949ddaecc8bf8e707b93f17.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429615 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429631 |     96257 | done8826e5af80dda71509c44b83f1853924.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429694 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |

but then some more got added during the archiving process:

| idinvalidation | idarchive | name                                         | idsite | date1      | date2      | period | ts_invalidated      | ts_started | status | report |

|        1430850 |      NULL | done.Funnels                                 |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1430851 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1430852 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1430853 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |

And those rows keep coming back. This is probably due to the invalidation cascading.

@diosmosis
Copy link
Member

Hi @nelhefni, I'll add context to your investigation below:

After the update, nothing was added to the archive_invalidations table, but 2 raws were removed:

This is expected, we remove pending invalidations for the old funnel ID.

Then, when the cron job started, the invalidation process added a lot of new raws to the archive_invalidations table:

So far, still expected: when we edit the funnel, we add the information to rearchive to list in the option table, core:archive will look at that list before archiving a site and create invalidations. We do this so the request to edit a funnel doesn't slow down due to inserting invalidations.

Not too sure why there are a lot of idarchive set to NULL though.

The idarchive is just a diagnostic, the code doesn't use it. It's set to the existing latest idarchive w/ the same parameters in case we want to know which previous idarchive the invalidation is meant to replace. If it's NULL, it would mean there wasn't one.

I'm not sure to understand the purpose of this done.Funnels, could you please explain the concept behind it?

So the archive in question is one of the old ones that are meant to be replaced. I'm not sure exactly what you're asking for, so I'll explain but feel free to ask if there's more information you want. The "done flag" of an archive tells us a couple things:

  • having done in the name tells us that this row is the status indicator for the archive
  • having .Funnels means the archive should only contain reports for the Funnels plugin. Archiving in matomo is the slowest and most intensive part, so we don't want to do it when we don't have to. If we know that we don't need to archive every other plugin, then we can save a lot of computation by only archiving the single one. If we set the value to DONE_PARTIAL (5) then we know it only has some or one report for the plugin that are meant to override older archives.

This was also a partial archive and should not have been set to DONE_INVALIDATED. This was an issue that I thought was fixed, but I don't see the change in 4.x-dev. I'll put up a PR for this, I suspect it will solve some problems.

And those raws keep coming back. This is probably due to the invalidation cascading.

That's odd, they shouldn't come back if you remove them from the invalidations table. Can you check the contents of the ReArchiveList option? (ie, SELECT * FROM option WHERE option_name = 'ReArchiveList')

@nelhefni
Copy link
Author

nelhefni commented Apr 8, 2021

@diosmosis thanks for the explanation

Here is the content of the ReArchiveList option:

SELECT * FROM matomo_option WHERE option_name = 'ReArchiveList'

| option_name   | option_value                                                                                                                                                                                                 | autoload |

| ReArchiveList | a:2:{i:0;s:86:"{"idSites":"all","pluginName":"Cohorts","report":null,"startDate":null,"segment":null}";i:1;s:88:"{"idSites":"all","pluginName":"UsersFlow","report":null,"startDate":null,"segment":null}";} |        0 |

This was referenced Apr 9, 2021
Merged
Merged
@diosmosis
Copy link
Member

Hi @nelhefni, thanks for going through the trouble of investigating this issue so thoroughly! I was able to track down the root causes of the two issues identified so far:

They will hopefully be ready to test or merge soon.

I'm still not sure about the Funnel invalidations reoccurring, though. If you continue to see this, it might help to add a warning log in ArchiveInvalidator::markArchivesInvalidated() when the parameters match the plugin/report/idsite combination and see where it's happening.

@nelhefni
Copy link
Author

nelhefni commented Apr 9, 2021

thanks @diosmosis !

@diosmosis
Copy link
Member

@nelhefni both of the PRs have been merged, would you be able to see if they solve the problem for your?

@diosmosis diosmosis mentioned this issue Apr 15, 2021
Closed
@diosmosis
Copy link
Member

Assuming this was all fixed in 4.3 as we're no issues so far. I'll close this.

@mattab mattab added the answered For when a question was asked and we referred to forum or answered it. label Jul 27, 2021
@tsteur tsteur removed the Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. label Sep 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
answered For when a question was asked and we referred to forum or answered it.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert partially pr 17216 nelhefni/matomo
4 participants
@diosmosis @tsteur @mattab @nelhefni