Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When a custom dimension is deleted, disable cron archiving for all segments that use this custom dimension #16152

Closed
mattab opened this issue Mar 12, 2018 · 4 comments · Fixed by #18383
Closed

When a custom dimension is deleted, disable cron archiving for all segments that use this custom dimension #16152

mattab opened this issue Mar 12, 2018 · 4 comments · Fixed by #18383
Assignees
@peterhashair
Labels
Bug For errors / faults / flaws / inconsistencies etc. c: Custom Dimensions For issues related to the Custom Dimensions plugin.
Milestone
4.8.0

Comments

@mattab
Copy link
Member

mattab commented Mar 12, 2018

Context

When a custom dimension is delete, and possibly when it's deactivated (to be tested), then any segment that still use this custom dimension, will fail to process.

  • When the segment is processed from cron, it leads to the whole archiving process to fail and error, preventing other reports to run. The error looks like: ERROR [2018-03-12 00:12:06] Got invalid response from API request: ?module=API&method=API.get&idSite=2&period=day&date=last52&format=php&trigger=archivephp. Response was 'a:2:{s:6:"result";s:5:"error";s:7:"message";s:81:"Segment 'dimension7' is not a supported segment. - caused by plugin CustomReports";}'
  • When the segment is processed via the UI only, the user would see the error message in the UI Segment 'dimension7' is not a supported segment. - caused by plugin CustomReports

Solution

The goal of this issue is to address the problem of the Cron archive only. Proposed solution:

  • Whenever a custom dimension is deleted (and likely also when it is de-activated), then we look for all segments that use this custom dimension
  • for each segment, we mark them as "autoArchive = false" (see SegmentEditor.add/update API parameter), which maps to auto_archive=0 in the segment table

As a result, the cron processing will not fail anymore, and all reports will be available, as expected.
@tsteur
Copy link
Member

tsteur commented Mar 12, 2018

FYI: Custom reports stores its segment definition in a custom reports table. So ideally this solution would trigger an event that lets other plugins know to modify all segments (this would be needed in custom dimensions, but maybe we can use some generic event so it also works in the future for other things maybe, not needed though), and offer a simple method to manipulate a segment (this would be needed in core), eg removeSegmentCondition(fullSegment, segmentNameToRemove).

@tsteur
Copy link
Member

tsteur commented Mar 12, 2018

BTW the segment archiving problem is more a general core problem and does not only apply to custom dimensions. Same problem exists when deactivating a plugin that provides segments etc. Maybe the segment filter in core should simply ignore such failures and leave a segment filter out when it is not available? Whether the segment is valid would then be only checked when editing or adding a segment etc.

@mattab
Copy link
Member Author

mattab commented Mar 12, 2018 •
edited by tsteur

👍 so to summarise the solution could be to do 2 separate changes:

  1. add a check in the core:archive command, where at the start of the process, or before triggering archiving for a particular segment, we would check if the segment will work. If not, it's skipped and a message displayed on screen (but not an error).
  2. additionally handle the "Custom reports" use case specifically, since the segment is built-in the custom report, and does not use the standard Segment feature (so the "check if the segment will work in core:archive" won't cover the custom reports segments). The custom reports plugin would need to handle this case specifically.
  3. Show in the UI when this segment is selected, that reports are no longer being processed because the segment is missing segment parts.

@tsteur tsteur transferred this issue from matomo-org/plugin-CustomDimensions Jun 29, 2020
@tsteur tsteur added Bug For errors / faults / flaws / inconsistencies etc. c: Custom Dimensions For issues related to the Custom Dimensions plugin. labels Jun 29, 2020
@tsteur tsteur added this to the 4.3.0 milestone Jun 29, 2020
@mattab
Copy link
Member Author

mattab commented Jan 5, 2021

we got this issue a few times over the last weeks, the last occurrence:

Segment 'dimension2' is not a supported segment.

Full error message below (reproduced on several instances):

ERROR [2021-01-05 00:48:47] 16145  Uncaught exception: /var/www/html/core/Segment.php(210): Segment 'dimension2' is not a supported segment.
Segment 'dimension2' is not a supported segment.



[Exception]
Segment 'dimension2' is not a supported segment.



Exception trace:
() at /var/www/html/core/Segment.php:210
Piwik\Segment->getSegmentByName() at /var/www/html/core/Segment.php:258
Piwik\Segment->getExpressionsWithUnionsResolved() at /var/www/html/core/Segment.php:236
Piwik\Segment->initializeSegment() at /var/www/html/core/Segment.php:163
Piwik\Segment->__construct() at /var/www/html/core/CronArchive.php:908
Piwik\CronArchive->invalidateWithSegments() at /var/www/html/core/CronArchive.php:879
Piwik\CronArchive->invalidateRecentDate() at /var/www/html/core/CronArchive.php:817
Piwik\CronArchive->invalidateArchivedReportsForSitesThatNeedToBeArchivedAgain() at /var/www/html/core/CronArchive/QueueConsumer.php:160
Piwik\CronArchive\QueueConsumer->getNextArchivesToProcess() at /var/www/html/core/CronArchive.php:372
Piwik\CronArchive->run() at /var/www/html/core/CronArchive.php:262
Piwik\CronArchive->Piwik\{closure}() at /var/www/html/core/Access.php:664
Piwik\Access::doAsSuperUser() at /var/www/html/core/CronArchive.php:268
Piwik\CronArchive->main() at /var/www/html/plugins/CoreConsole/Commands/CoreArchiver.php:27
Piwik\Plugins\CoreConsole\Commands\CoreArchiver->execute() at /var/www/html/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:257
Symfony\Component\Console\Command\Command->run() at /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php:874
Symfony\Component\Console\Application->doRunCommand() at /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php:195
Symfony\Component\Console\Application->doRun() at n/a:n/a
call_user_func() at /var/www/html/core/Console.php:130
Piwik\Console->Piwik\{closure}() at /var/www/html/core/Access.php:673
Piwik\Access::doAsSuperUser() at /var/www/html/core/Console.php:131
Piwik\Console->doRunImpl() at /var/www/html/core/Console.php:82
Piwik\Console->doRun() at /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php:126
Symfony\Component\Console\Application->run() at /var/www/html/console:32

@peterhashair peterhashair self-assigned this Nov 18, 2021
@peterhashair peterhashair mentioned this issue Nov 28, 2021
Merged
11 tasks
@peterhashair peterhashair linked a pull request Nov 28, 2021 that will close this issue
disable cron task when segment is deleted #18383
Merged
11 tasks
@tsteur tsteur mentioned this issue Dec 2, 2021
Open
@justinvelluppillai justinvelluppillai modified the milestones: 4.7.0, 4.8.0 Jan 18, 2022
@tsteur tsteur mentioned this issue Nov 8, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@peterhashair peterhashair

Labels
Bug For errors / faults / flaws / inconsistencies etc. c: Custom Dimensions For issues related to the Custom Dimensions plugin.
Projects
None yet
Milestone
4.8.0
Development

Successfully merging a pull request may close this issue.

disable cron task when segment is deleted matomo-org/matomo
4 participants
@tsteur @mattab @justinvelluppillai @peterhashair