When a custom dimension is deleted, disable cron archiving for all segments that use this custom dimension #16152

Context

When a custom dimension is delete, and possibly when it's deactivated (to be tested), then any segment that still use this custom dimension, will fail to process.

• When the segment is processed from cron, it leads to the whole archiving process to fail and error, preventing other reports to run. The error looks like: ERROR [2018-03-12 00:12:06] Got invalid response from API request: ?module=API&method=API.get&idSite=2&period=day&date=last52&format=php&trigger=archivephp. Response was 'a:2:{s:6:"result";s:5:"error";s:7:"message";s:81:"Segment 'dimension7' is not a supported segment. - caused by plugin CustomReports";}'
• When the segment is processed via the UI only, the user would see the error message in the UI Segment 'dimension7' is not a supported segment. - caused by plugin CustomReports

Solution

The goal of this issue is to address the problem of the Cron archive only. Proposed solution:

• Whenever a custom dimension is deleted (and likely also when it is de-activated), then we look for all segments that use this custom dimension
• for each segment, we mark them as "autoArchive = false" (see SegmentEditor.add/update API parameter), which maps to auto_archive=0 in the segment table

As a result, the cron processing will not fail anymore, and all reports will be available, as expected.

FYI: Custom reports stores its segment definition in a custom reports table. So ideally this solution would trigger an event that lets other plugins know to modify all segments (this would be needed in custom dimensions, but maybe we can use some generic event so it also works in the future for other things maybe, not needed though), and offer a simple method to manipulate a segment (this would be needed in core), eg removeSegmentCondition(fullSegment, segmentNameToRemove).

BTW the segment archiving problem is more a general core problem and does not only apply to custom dimensions. Same problem exists when deactivating a plugin that provides segments etc. Maybe the segment filter in core should simply ignore such failures and leave a segment filter out when it is not available? Whether the segment is valid would then be only checked when editing or adding a segment etc.

:+1: so to summarise the solution could be to do 2 separate changes:

1. add a check in the core:archive command, where at the start of the process, or before triggering archiving for a particular segment, we would check if the segment will work. If not, it's skipped and a message displayed on screen (but not an error).
2. additionally handle the "Custom reports" use case specifically, since the segment is built-in the custom report, and does not use the standard Segment feature (so the "check if the segment will work in core:archive" won't cover the custom reports segments). The custom reports plugin would need to handle this case specifically.
3. Show in the UI when this segment is selected, that reports are no longer being processed because the segment is missing segment parts.

we got this issue a few times over the last weeks, the last occurrence:

Segment 'dimension2' is not a supported segment.

Full error message below (reproduced on several instances):

ERROR [2021-01-05 00:48:47] 16145  Uncaught exception: /var/www/html/core/Segment.php(210): Segment 'dimension2' is not a supported segment.
Segment 'dimension2' is not a supported segment.

[Exception]
Segment 'dimension2' is not a supported segment.

Exception trace:
() at /var/www/html/core/Segment.php:210
Piwik\Segment->getSegmentByName() at /var/www/html/core/Segment.php:258
Piwik\Segment->getExpressionsWithUnionsResolved() at /var/www/html/core/Segment.php:236
Piwik\Segment->initializeSegment() at /var/www/html/core/Segment.php:163
Piwik\Segment->__construct() at /var/www/html/core/CronArchive.php:908
Piwik\CronArchive->invalidateWithSegments() at /var/www/html/core/CronArchive.php:879
Piwik\CronArchive->invalidateRecentDate() at /var/www/html/core/CronArchive.php:817
Piwik\CronArchive->invalidateArchivedReportsForSitesThatNeedToBeArchivedAgain() at /var/www/html/core/CronArchive/QueueConsumer.php:160
Piwik\CronArchive\QueueConsumer->getNextArchivesToProcess() at /var/www/html/core/CronArchive.php:372
Piwik\CronArchive->run() at /var/www/html/core/CronArchive.php:262
Piwik\CronArchive->Piwik\{closure}() at /var/www/html/core/Access.php:664
Piwik\Access::doAsSuperUser() at /var/www/html/core/CronArchive.php:268
Piwik\CronArchive->main() at /var/www/html/plugins/CoreConsole/Commands/CoreArchiver.php:27
Piwik\Plugins\CoreConsole\Commands\CoreArchiver->execute() at /var/www/html/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:257
Symfony\Component\Console\Command\Command->run() at /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php:874
Symfony\Component\Console\Application->doRunCommand() at /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php:195
Symfony\Component\Console\Application->doRun() at n/a:n/a
call_user_func() at /var/www/html/core/Console.php:130
Piwik\Console->Piwik\{closure}() at /var/www/html/core/Access.php:673
Piwik\Access::doAsSuperUser() at /var/www/html/core/Console.php:131
Piwik\Console->doRunImpl() at /var/www/html/core/Console.php:82
Piwik\Console->doRun() at /var/www/html/vendor/symfony/console/Symfony/Component/Console/Application.php:126
Symfony\Component\Console\Application->run() at /var/www/html/console:32