@nelhefni opened this Issue on April 7th 2021

Hello,

We think PR https://github.com/matomo-org/matomo/pull/17216 has introduced some regressions to the archiving process since Matomo 4.2.0

FYI, browser archiving is disabled:

enable_browser_archiving_triggering = 0
browser_archiving_disabled_enforce = 1
archiving_range_force_on_browser_request = 0

For a given plugin, we are facing 2 issues since the update to Matomo 4.2.0,

  • Issue#1: Some daily reports disappear from the UI, even though delete_reports_enable is set to 0
  • Issue#2: Some weekly reports have inaccurate data: gather the values of only some days of the week.

This seems to be related to the changes made with the handling of partial archives.
Indeed, we have a lot of partial archives in our DB, which doesn't contain the reports of our impacted plugins:

MariaDB [matomo]> select * from matomo_archive_numeric_2021_03 where idarchive = 46176;

| idarchive | name        | idsite   | date1      | date2      | period    | ts_archived         | value    |

|     46176 | myPlugin    |        1 | 2021-03-22 | 2021-03-22 |         1 | 2021-03-23 00:05:52 |       10 |
|     46176 | otherPlugin |        1 | 2021-03-22 | 2021-03-22 |         1 | 2021-03-23 00:05:52 |       20 |
|     46176 | done        |        1 | 2021-03-22 | 2021-03-22 |         1 | 2021-03-23 00:05:52 |        1 |
MariaDB [matomo]> select * from matomo_archive_numeric_2021_03 where idarchive = 56405;

| idarchive | name        | idsite   | date1      | date2      | period    | ts_archived         | value    |

|     56405 | otherPlugin |        1 | 2021-03-22 | 2021-03-22 |         1 | 2021-04-05 18:52:03 |       20 |
|     56405 | done        |        1 | 2021-03-22 | 2021-03-22 |         1 | 2021-04-05 18:52:03 |        5 |

As you can see, myPlugin report is not part of partial archive 56405 (partial because the value of done row is 5), but it is nevertheless selected since Matomo 4.2.0, as ts_archived is more recent.

Since https://github.com/matomo-org/matomo/pull/17216, checkAuthorizedToArchive = false is passed to Rules::getSelectableDoneFlagValues during the retrival of archive to display in the UI
https://github.com/matomo-org/matomo/blob/2c30dc8d231af58cdf2516a413da2d86bf7d62c6/core/DataAccess/ArchiveSelector.php#L370

As a consequence, both ArchiveWriter::DONE_PARTIAL and ArchiveWriter::DONE_INVALIDATED are added to the list of accepted value. Which explains why archive 56405 is selected instead of 46176 in our case.
https://github.com/matomo-org/matomo/blob/2c30dc8d231af58cdf2516a413da2d86bf7d62c6/core/ArchiveProcessor/Rules.php#L321

Thus, as soon as archive 56405 was created, myPlugin report disappeared from the UI.
I don't think ArchiveWriter::DONE_PARTIAL should be part of the list of accepted values.
We have been able to fix this issue#1 with this PR https://github.com/matomo-org/matomo/pull/17425

Concerning issue#2, we have been able to fix it by reverting the changes done in the same PR to ArchiveSelector::findArchiveDataWithLatestTsArchived, and then invalidating faulty reports and re-running the cron job.

Your Environment

Matomo 4.2.1

@diosmosis commented on April 7th 2021 Member

Hi @nelhefni, thanks for the detailed report. Partial archives are meant to be included in the entire list of all idarchives to look through. We don't select only one archive, but the latest DONE_OK or DONE_INVALIDATED archive, then all DONE_PARTIAL archives that are later. We then look for data in all of those archives. Using this knowledge, can you take another look? Or if possible can you provide access so we can take a closer look?

@nelhefni commented on April 8th 2021

Hi @diosmosis, thanks for the fast answer.
I've been tracing the code, and ArchiveSelector::getArchiveIds returns only one idarchive

https://github.com/matomo-org/matomo/blob/2c30dc8d231af58cdf2516a413da2d86bf7d62c6/core/DataAccess/ArchiveSelector.php#L198

archiveIds = 
Array
(
  [0] => Array
  (
    [idsite] => 1
    [date1] => 2021-03-22
    [date2] => 2021-03-22
    [archives] =>
    56405|done|5,53884|done|5,49890|done.Funnels|5,46176|done|1
  )
)

As there is only one element in $idarchives array, we loop only once in https://github.com/matomo-org/matomo/blob/2c30dc8d231af58cdf2516a413da2d86bf7d62c6/core/DataAccess/ArchiveSelector.php#L202

then, https://github.com/matomo-org/matomo/blob/2c30dc8d231af58cdf2516a413da2d86bf7d62c6/core/DataAccess/ArchiveSelector.php#L206

$pairs = 
Array
(
  [0] => 56405|done|5
  [1] => 53884|done|5
  [2] => 49890|done.Funnels|5
  [3] => 46176|done|1
)

then we break, because idarchive 56405 contains done without any dot https://github.com/matomo-org/matomo/blob/2c30dc8d231af58cdf2516a413da2d86bf7d62c6/core/DataAccess/ArchiveSelector.php#L212

finally, we have only one idarchive returned https://github.com/matomo-org/matomo/blob/2c30dc8d231af58cdf2516a413da2d86bf7d62c6/core/Archive.php#L638

$idarchivesByReport =
Array
(
  [done] => Array
  (
    [2021-03-22,2021-03-22] => Array
    (
      [0] => 56405
    )
  )
)
@diosmosis commented on April 8th 2021 Member

@nelhefni thanks again for the detailed post. That code looks broken but it's actually working correctly, the problem is that archives like 56405|done|5 should never exist... partial archives have to be specific to a plugin, they should never have a done flag of just done.

Can you see what reports are in 56405 (eg, what names are in the blob table for 56405)?

I think we can put in some code making sure partial archives are never saved w/ that done flag, and modify the code you reference just in case this edge case happens in some other way, but I can't think of how this might happen.

@nelhefni commented on April 8th 2021

@diosmosis thanks for the input, the issue could then be a side effect of https://github.com/matomo-org/matomo/issues/17434

Here are the reports for idarchive 56405:


MariaDB [matomo]> select * from matomo_archive_numeric_2021_03 where idarchive = 56405;

| idarchive | name                               | idsite | date1      | date2      | period | ts_archived         | value   |

|    56405 | Actions_nb_outlinks                |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Actions_nb_pageviews               |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Actions_nb_uniq_outlinks           |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Actions_nb_uniq_pageviews          |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | bounce_count                       |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | done                               |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |        5 |
|    56405 | Goal_10_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_10_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_10_revenue                    |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |     99.9 |
|    56405 | Goal_11_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_11_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_12_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_12_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_13_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_13_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_14_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_14_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_15_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_15_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_16_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_16_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_17_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_17_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_18_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_18_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_19_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_19_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_1_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_1_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_20_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_20_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_21_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_21_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_22_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_22_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_23_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_23_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_24_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_24_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_25_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_25_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_26_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_26_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_27_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_27_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_28_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_28_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_29_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_29_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_2_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_2_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_30_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_30_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_31_nb_conversions             |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_31_nb_visits_converted        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_3_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_3_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_4_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_4_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_5_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_5_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_6_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_6_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_7_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_7_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_8_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_8_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_9_nb_conversions              |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_9_nb_visits_converted         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Goal_nb_conversions                |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_nb_visits_converted           |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Goal_revenue                       |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |     99.9 |
|    56405 | max_actions                        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | nb_actions                         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | nb_uniq_visitors                   |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | nb_users                           |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | nb_visits                          |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | nb_visits_converted                |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | PagePerformance_domcompletion_hits |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_domcompletion_time |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_domprocessing_hits |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_domprocessing_time |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_network_hits       |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_network_time       |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_onload_hits        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_onload_time        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_pageload_hits      |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_pageload_time      |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_servery_time       |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_server_hits        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_transfer_hits      |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | PagePerformance_transfer_time      |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |      999 |
|    56405 | Referrers_distinctCampaigns        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Referrers_distinctKeywords         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Referrers_distinctSearchEngines    |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Referrers_distinctSocialNetworks   |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Referrers_distinctWebsites         |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | Referrers_distinctWebsitesUrls     |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | sum_visit_length                   |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:03 |      999 |
|    56405 | UserCountry_distinctCountries      |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-05 18:52:14 |        1 |

Which doesn't include myPlugin

@diosmosis commented on April 8th 2021 Member

@nelhefni the Goals plugin doesn't have any logic to rearchive in the past when it changes: https://github.com/matomo-org/matomo/blob/4.x-dev/plugins/Goals/API.php#L198-L232. The funnels plugin does, which is connected, but then I would expect to see Funnels reports in this table. Can you try editing a goal or creating a goal, then checking what gets added to archive_invalidations (then truncate the table or take note of the max idinvalidation and delete everything greater than it to avoid launching any archiving)?

Based on the archive contents above it seems like that archive should have a done value of DONE_OK (or 1). I guess it might be possible that because it gets set as value = 5, it thinks the archiving never happens and keeps trying... I'll have a look at the code when I can to see where it might be possible to set the value as 5 for an archive w/ name = 'done'.

@nelhefni commented on April 8th 2021

@diosmosis I did update the funnel attached to one of my goals.
ID of the funnel changed from 32 to 33.
After the update, nothing was added to the archive_invalidations table, but 2 rows were removed:

| idinvalidation | idarchive | name                                                | idsite | date1      | date2      | period | ts_invalidated      | ts_started          | status | report |

|        1260359 |    457962 | done.Funnels                                        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-03-25 06:06:09 | 2021-03-26 20:04:28 |      0 | 32     |
|        1260644 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-03-25 06:06:49 | NULL                |      0 | 32     |

Then, when the cron job started, the invalidation process added a lot of new rows to the archive_invalidations table:

select * from matomo_archive_invalidations where name like '%unnel%' and report = 33;

| idinvalidation | idarchive | name                                                | idsite | date1      | date2      | period | ts_invalidated      | ts_started | status | report |

|        1429392 |     54863 | done.Funnels                                        |      1 | 2021-03-01 | 2021-03-01 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429393 |     54894 | done.Funnels                                        |      1 | 2021-03-01 | 2021-03-07 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429394 |    190726 | done.Funnels                                        |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429395 |     54862 | done.Funnels                                        |      1 | 2021-03-02 | 2021-03-02 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429396 |     54858 | done.Funnels                                        |      1 | 2021-03-03 | 2021-03-03 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429397 |     54827 | done.Funnels                                        |      1 | 2021-03-04 | 2021-03-04 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429398 |     54825 | done.Funnels                                        |      1 | 2021-03-05 | 2021-03-05 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429399 |     54822 | done.Funnels                                        |      1 | 2021-03-06 | 2021-03-06 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429400 |     54447 | done.Funnels                                        |      1 | 2021-03-07 | 2021-03-07 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429401 |     54446 | done.Funnels                                        |      1 | 2021-03-08 | 2021-03-08 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429402 |     54472 | done.Funnels                                        |      1 | 2021-03-08 | 2021-03-14 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429403 |     54443 | done.Funnels                                        |      1 | 2021-03-09 | 2021-03-09 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429404 |     54411 | done.Funnels                                        |      1 | 2021-03-10 | 2021-03-10 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429405 |     54410 | done.Funnels                                        |      1 | 2021-03-11 | 2021-03-11 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429406 |     54407 | done.Funnels                                        |      1 | 2021-03-12 | 2021-03-12 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429407 |     54019 | done.Funnels                                        |      1 | 2021-03-13 | 2021-03-13 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429408 |     54015 | done.Funnels                                        |      1 | 2021-03-14 | 2021-03-14 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429409 |     53983 | done.Funnels                                        |      1 | 2021-03-15 | 2021-03-15 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429410 |     54017 | done.Funnels                                        |      1 | 2021-03-15 | 2021-03-21 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429411 |     53982 | done.Funnels                                        |      1 | 2021-03-16 | 2021-03-16 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429412 |     53978 | done.Funnels                                        |      1 | 2021-03-17 | 2021-03-17 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429413 |     53947 | done.Funnels                                        |      1 | 2021-03-18 | 2021-03-18 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429414 |     53946 | done.Funnels                                        |      1 | 2021-03-19 | 2021-03-19 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429415 |     53942 | done.Funnels                                        |      1 | 2021-03-20 | 2021-03-20 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429416 |     53894 | done.Funnels                                        |      1 | 2021-03-21 | 2021-03-21 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429417 |     53890 | done.Funnels                                        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429418 |     67174 | done.Funnels                                        |      1 | 2021-03-22 | 2021-03-28 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429419 |     53889 | done.Funnels                                        |      1 | 2021-03-23 | 2021-03-23 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429420 |      NULL | done.Funnels                                        |      1 | 2021-03-24 | 2021-03-24 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429421 |      NULL | done.Funnels                                        |      1 | 2021-03-25 | 2021-03-25 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429422 |      NULL | done.Funnels                                        |      1 | 2021-03-26 | 2021-03-26 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429423 |      NULL | done.Funnels                                        |      1 | 2021-03-27 | 2021-03-27 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429424 |      NULL | done.Funnels                                        |      1 | 2021-03-28 | 2021-03-28 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429425 |      NULL | done.Funnels                                        |      1 | 2021-03-29 | 2021-03-29 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429426 |      NULL | done.Funnels                                        |      1 | 2021-03-29 | 2021-04-04 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429427 |      NULL | done.Funnels                                        |      1 | 2021-03-30 | 2021-03-30 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429428 |      NULL | done.Funnels                                        |      1 | 2021-03-31 | 2021-03-31 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429455 |    457962 | done.Funnels                                        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429456 |      NULL | done.Funnels                                        |      1 | 2021-04-01 | 2021-04-01 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429457 |      NULL | done.Funnels                                        |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429458 |      NULL | done.Funnels                                        |      1 | 2021-04-02 | 2021-04-02 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429459 |      NULL | done.Funnels                                        |      1 | 2021-04-03 | 2021-04-03 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429460 |      NULL | done.Funnels                                        |      1 | 2021-04-04 | 2021-04-04 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429461 |      NULL | done.Funnels                                        |      1 | 2021-04-05 | 2021-04-05 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429462 |      NULL | done.Funnels                                        |      1 | 2021-04-05 | 2021-04-11 |      2 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429463 |      NULL | done.Funnels                                        |      1 | 2021-04-06 | 2021-04-06 |      1 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429471 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-01 | 2021-03-01 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429472 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-01 | 2021-03-07 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429473 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429474 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-02 | 2021-03-02 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429475 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-03 | 2021-03-03 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429476 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-04 | 2021-03-04 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429477 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-05 | 2021-03-05 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429478 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-06 | 2021-03-06 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429479 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-07 | 2021-03-07 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429480 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-08 | 2021-03-08 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429481 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-08 | 2021-03-14 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429482 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-09 | 2021-03-09 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429483 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-10 | 2021-03-10 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429484 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-11 | 2021-03-11 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429485 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-12 | 2021-03-12 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429486 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-13 | 2021-03-13 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429487 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-14 | 2021-03-14 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429488 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-15 | 2021-03-15 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429489 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-15 | 2021-03-21 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429490 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-16 | 2021-03-16 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429491 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-17 | 2021-03-17 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429492 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-18 | 2021-03-18 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429493 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-19 | 2021-03-19 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429494 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-20 | 2021-03-20 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429495 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-21 | 2021-03-21 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429496 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429497 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-22 | 2021-03-28 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429498 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-23 | 2021-03-23 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429499 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-24 | 2021-03-24 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429500 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-25 | 2021-03-25 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429501 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-26 | 2021-03-26 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429502 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-27 | 2021-03-27 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429503 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-28 | 2021-03-28 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429504 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-29 | 2021-03-29 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429505 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-29 | 2021-04-04 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429506 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-30 | 2021-03-30 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429507 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-03-31 | 2021-03-31 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429534 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429535 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-01 | 2021-04-01 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429536 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429537 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-02 | 2021-04-02 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429538 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-03 | 2021-04-03 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429539 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-04 | 2021-04-04 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429540 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-05 | 2021-04-05 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429541 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-05 | 2021-04-11 |      2 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429542 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels        |      1 | 2021-04-06 | 2021-04-06 |      1 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429550 |     54888 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-01 | 2021-03-01 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429551 |     55095 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-01 | 2021-03-07 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429552 |     96258 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429553 |     54879 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-02 | 2021-03-02 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429554 |     54868 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-03 | 2021-03-03 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429555 |     54853 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-04 | 2021-03-04 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429556 |     54844 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-05 | 2021-03-05 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429557 |     54831 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-06 | 2021-03-06 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429558 |     54672 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-07 | 2021-03-07 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429559 |     54464 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-08 | 2021-03-08 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429560 |     54677 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-08 | 2021-03-14 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429561 |     54452 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-09 | 2021-03-09 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429562 |     54437 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-10 | 2021-03-10 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429563 |     54428 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-11 | 2021-03-11 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429564 |     54416 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-12 | 2021-03-12 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429565 |     54401 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-13 | 2021-03-13 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429566 |     54390 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-14 | 2021-03-14 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429567 |     54008 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-15 | 2021-03-15 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429568 |     54223 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-15 | 2021-03-21 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429569 |     54000 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-16 | 2021-03-16 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429570 |     53988 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-17 | 2021-03-17 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429571 |     53973 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-18 | 2021-03-18 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429572 |     53964 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-19 | 2021-03-19 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429573 |     53952 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-20 | 2021-03-20 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429574 |     53936 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-21 | 2021-03-21 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429575 |     53907 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429576 |     67373 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-22 | 2021-03-28 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429577 |     53898 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-23 | 2021-03-23 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429578 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-24 | 2021-03-24 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429579 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-25 | 2021-03-25 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429580 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-26 | 2021-03-26 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429581 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-27 | 2021-03-27 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429582 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-28 | 2021-03-28 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429583 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-29 | 2021-03-29 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429584 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-29 | 2021-04-04 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429585 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-30 | 2021-03-30 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429586 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-03-31 | 2021-03-31 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429613 |    891194 | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429614 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-01 | 2021-04-01 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429615 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429616 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-02 | 2021-04-02 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429617 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-03 | 2021-04-03 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429618 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-04 | 2021-04-04 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429619 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-05 | 2021-04-05 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429620 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-05 | 2021-04-11 |      2 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429621 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels        |      1 | 2021-04-06 | 2021-04-06 |      1 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429629 |     54887 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-01 | 2021-03-01 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429630 |     55096 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-01 | 2021-03-07 |      2 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429631 |     96257 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429632 |     54878 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-02 | 2021-03-02 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429633 |     54867 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-03 | 2021-03-03 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429634 |     54852 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-04 | 2021-03-04 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429635 |     54843 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-05 | 2021-03-05 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429636 |     54832 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-06 | 2021-03-06 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429637 |     54475 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-07 | 2021-03-07 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429638 |     54463 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-08 | 2021-03-08 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429639 |     54678 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-08 | 2021-03-14 |      2 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429640 |     54451 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-09 | 2021-03-09 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429641 |     54436 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-10 | 2021-03-10 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429642 |     54427 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-11 | 2021-03-11 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429643 |     54415 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-12 | 2021-03-12 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429644 |     54400 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-13 | 2021-03-13 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429645 |     54389 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-14 | 2021-03-14 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429646 |     54009 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-15 | 2021-03-15 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429647 |     54222 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-15 | 2021-03-21 |      2 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429648 |     53999 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-16 | 2021-03-16 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429649 |     53987 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-17 | 2021-03-17 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429650 |     53972 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-18 | 2021-03-18 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429651 |     53963 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-19 | 2021-03-19 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429652 |     53951 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-20 | 2021-03-20 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429653 |     53937 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-21 | 2021-03-21 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429654 |     53908 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-22 | 2021-03-22 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429655 |     67372 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-22 | 2021-03-28 |      2 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429656 |     53899 | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-23 | 2021-03-23 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429657 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-24 | 2021-03-24 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429658 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-25 | 2021-03-25 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429659 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-26 | 2021-03-26 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429660 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-27 | 2021-03-27 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429661 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-28 | 2021-03-28 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429662 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-29 | 2021-03-29 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429663 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-29 | 2021-04-04 |      2 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429664 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-30 | 2021-03-30 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429665 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-03-31 | 2021-03-31 |      1 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429692 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-01-01 | 2021-12-31 |      4 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429693 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-01 | 2021-04-01 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429694 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429695 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-02 | 2021-04-02 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429696 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-03 | 2021-04-03 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429697 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-04 | 2021-04-04 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429698 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-05 | 2021-04-05 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429699 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-05 | 2021-04-11 |      2 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |
|        1429700 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels        |      1 | 2021-04-06 | 2021-04-06 |      1 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |

This seems to make sense, as we have 3 segments, and go back to March 1st because of the following setting:
rearchive_reports_in_past_last_n_months = last1
Not too sure why there are a lot of idarchive set to NULL though.

Moreover, If I look for the corresponding idarchive in matomo_archive_numeric2021**, I find only one row:

select * from matomo_archive_numeric_2021_03 where idarchive = 54863;

| idarchive | name         | idsite | date1      | date2      | period | ts_archived         | value |

|     54863 | done.Funnels |      1 | 2021-03-01 | 2021-03-01 |      1 | 2021-03-25 06:35:35 |     4 |

I'm not sure to understand the purpose of this done.Funnels, could you please explain the concept behind it?

Also, I deleted manually all the rows for period 3, as I want to avoid the issue described here: https://github.com/matomo-org/matomo/issues/17434,

delete from matomo_archive_invalidations where name like '%unnel%' and period = 3 and report = 33;

| idinvalidation | idarchive | name                                         | idsite | date1      | date2      | period | ts_invalidated      | ts_started | status | report |

|        1429394 |    190726 | done.Funnels                                 |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429457 |      NULL | done.Funnels                                 |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1429473 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429536 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |
|        1429552 |     96258 | done0454a4912949ddaecc8bf8e707b93f17.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429615 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1429631 |     96257 | done8826e5af80dda71509c44b83f1853924.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1429694 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels |      1 | 2021-04-01 | 2021-04-30 |      3 | 2021-04-08 05:28:18 | NULL       |      0 | 33     |

but then some more got added during the archiving process:

| idinvalidation | idarchive | name                                         | idsite | date1      | date2      | period | ts_invalidated      | ts_started | status | report |

|        1430850 |      NULL | done.Funnels                                 |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:02 | NULL       |      0 | 33     |
|        1430851 |      NULL | done8826e5af80dda71509c44b83f1853924.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:17 | NULL       |      0 | 33     |
|        1430852 |      NULL | done0454a4912949ddaecc8bf8e707b93f17.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:12 | NULL       |      0 | 33     |
|        1430853 |      NULL | done252e705405118c93ee8ffcda1daff4ab.Funnels |      1 | 2021-03-01 | 2021-03-31 |      3 | 2021-04-08 05:28:07 | NULL       |      0 | 33     |

And those rows keep coming back. This is probably due to the invalidation cascading.

@diosmosis commented on April 8th 2021 Member

Hi @nelhefni, I'll add context to your investigation below:

After the update, nothing was added to the archive_invalidations table, but 2 raws were removed:

This is expected, we remove pending invalidations for the old funnel ID.

Then, when the cron job started, the invalidation process added a lot of new raws to the archive_invalidations table:

So far, still expected: when we edit the funnel, we add the information to rearchive to list in the option table, core:archive will look at that list before archiving a site and create invalidations. We do this so the request to edit a funnel doesn't slow down due to inserting invalidations.

Not too sure why there are a lot of idarchive set to NULL though.

The idarchive is just a diagnostic, the code doesn't use it. It's set to the existing latest idarchive w/ the same parameters in case we want to know which previous idarchive the invalidation is meant to replace. If it's NULL, it would mean there wasn't one.

I'm not sure to understand the purpose of this done.Funnels, could you please explain the concept behind it?

So the archive in question is one of the old ones that are meant to be replaced. I'm not sure exactly what you're asking for, so I'll explain but feel free to ask if there's more information you want. The "done flag" of an archive tells us a couple things:

  • having done in the name tells us that this row is the status indicator for the archive
  • having .Funnels means the archive should only contain reports for the Funnels plugin. Archiving in matomo is the slowest and most intensive part, so we don't want to do it when we don't have to. If we know that we don't need to archive every other plugin, then we can save a lot of computation by only archiving the single one. If we set the value to DONE_PARTIAL (5) then we know it only has some or one report for the plugin that are meant to override older archives.

This was also a partial archive and should not have been set to DONE_INVALIDATED. This was an issue that I thought was fixed, but I don't see the change in 4.x-dev. I'll put up a PR for this, I suspect it will solve some problems.

And those raws keep coming back. This is probably due to the invalidation cascading.

That's odd, they shouldn't come back if you remove them from the invalidations table. Can you check the contents of the ReArchiveList option? (ie, SELECT * FROM option WHERE option_name = 'ReArchiveList')

@nelhefni commented on April 8th 2021

@diosmosis thanks for the explanation

Here is the content of the ReArchiveList option:

SELECT * FROM matomo_option WHERE option_name = 'ReArchiveList'

| option_name   | option_value                                                                                                                                                                                                 | autoload |

| ReArchiveList | a:2:{i:0;s:86:"{"idSites":"all","pluginName":"Cohorts","report":null,"startDate":null,"segment":null}";i:1;s:88:"{"idSites":"all","pluginName":"UsersFlow","report":null,"startDate":null,"segment":null}";} |        0 |
@diosmosis commented on April 9th 2021 Member

Hi @nelhefni, thanks for going through the trouble of investigating this issue so thoroughly! I was able to track down the root causes of the two issues identified so far:

They will hopefully be ready to test or merge soon.

I'm still not sure about the Funnel invalidations reoccurring, though. If you continue to see this, it might help to add a warning log in ArchiveInvalidator::markArchivesInvalidated() when the parameters match the plugin/report/idsite combination and see where it's happening.

@nelhefni commented on April 9th 2021

thanks @diosmosis !

@diosmosis commented on April 12th 2021 Member

@nelhefni both of the PRs have been merged, would you be able to see if they solve the problem for your?

@diosmosis commented on June 4th 2021 Member

Assuming this was all fixed in 4.3 as we're no issues so far. I'll close this.

This Issue was closed on June 4th 2021
Powered by GitHub Issue Mirror