Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

three archiving fixes #16548

Merged
merged 9 commits into from Oct 9, 2020
Merged

three archiving fixes #16548

merged 9 commits into from Oct 9, 2020

Conversation

diosmosis
Copy link
Member

@diosmosis diosmosis commented Oct 8, 2020 •
edited

Changes:

  • unset status on failed archiving jobs so they will be picked up again in the next core:archive run

  • make sure ArchiveSelector::getArchiveIds behaves consistently with ArchiveSelector::getArchiveIdAndVisits

  • make sure the 'done' all plugins archives aren't used w/ pluginOnly=1, otherwise we will see the 'done' archive and assume we can use it. pluginOnly=1 is used when an invalidation for a specific plugin/report is added (ie, w/ reArchiveReport). so if it's set, and we find a usable archive, we force archiving again.

    NOTE: this should only happen in core:archive when we see an invalidation w/ the plugin column set. I don't think this can currently happen outside of ArchiveInvalidator::reArchiveReport().

FYI @tsteur

@diosmosis diosmosis added not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. Needs Review PRs that need a code review labels Oct 8, 2020
@diosmosis diosmosis added this to the 4.0.0-RC milestone Oct 8, 2020
@tsteur
Copy link
Member

tsteur commented Oct 8, 2020

👍

@diosmosis diosmosis changed the title two archiving fixes three archiving fixes Oct 9, 2020
@diosmosis
Copy link
Member Author

@tsteur found another (potential) issue fixed in a075a9b

code's a bit confusing, but I don't think it can be made much simpler (we need to get the latest idarchives for each requested plugin, then ignore older ones, and ignore the all plugins archive if individual plugin archives are found). this should avoid some odd edge cases. will merge if the build passes, but could use a review after the fact.

@diosmosis
Copy link
Member Author

@tsteur here's an example of the query, btw:

SELECT idsite, date1, date2,
        GROUP_CONCAT(CONCAT(idarchive,'|',`name`) ORDER BY idarchive DESC SEPARATOR ',') AS archives
   FROM matomo_archive_numeric_2014_05
  WHERE idsite IN (1)
    AND ((name IN ('done','done.ExamplePlugin')) AND (value IN (1,3,4,5)))
    AND ts_archived IS NOT NULL
    AND ((period = 1 AND date1 = '2020-10-03' AND date2 = '2020-10-03'))
GROUP BY idsite, date1, date2

the explain for it is:

--------+------------------------------------+
| id | select_type | table                          | partitions | type  | possible_keys                                   | key                       | key_len | ref  | rows | filtered | Extra                              |
+----+-------------+--------------------------------+------------+-------+-------------------------------------------------+---------------------------+---------+------+------+----------+------------------------------------+
|  1 | SIMPLE      | matomo_archive_numeric_2014_05 | NULL       | range | index_idsite_dates_period,index_period_archived | index_idsite_dates_period | 21      | NULL |    1 |     8.00 | Using index condition; Using where |
+----+-------------+--------------------------------+------------+-------+-------------------------------------------------+---------------------------+---------+------+------+----------+------------------------------------+

@diosmosis diosmosis merged commit 123be36 into 4.x-dev Oct 9, 2020
@diosmosis diosmosis deleted the archiving-fixes branch October 9, 2020 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Needs Review PRs that need a code review not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Projects
None yet
Milestone
4.0.0-RC
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@diosmosis @tsteur