This plugin's system tests shouldn't fail when new Segments are added in core #16140

Reproduce

• a new segment is added to Piwik core
• this plugin's system tests run
• and fail: <a href='/1'>#1</a>: Piwik\Plugins\CustomDimensions\tests\System\ApiTest: Differences with expected in '/home/travis/build/piwik/plugin-CustomDimensions/piwik/plugins/CustomDimensions/tests/System/processed/test___API.getSegmentsMetadata.xml' because a new segment was added in core

  Example

this failing build https://travis-ci.org/piwik/plugin-CustomDimensions/jobs/106380038

Solution

the system test plugins/CustomDimensions/tests/System/processed/test___API.getSegmentsMetadata.xml should only test for those segments that are added by the plugin.

Outcome

the build for CustomDimensions will fail less often and make us more agile and move faster without breaking builds.

As it is a system test it should kind of is supposed to fail and it's good when it's fail. That's the point of system tests. The test could be rewritten to an integration test but then we don't know if it actually works. I think it would be better to rather create better tools to fix system tests more easily

I think it would be better to rather create better tools to fix system tests more easily

Correct & wise thought :+1: