@mattab opened this Issue on February 3rd 2016 Member

Reproduce

  • a new segment is added to Piwik core
  • this plugin's system tests run
  • and fail: <a href='/1'>#1</a>: Piwik\Plugins\CustomDimensions\tests\System\ApiTest: Differences with expected in '/home/travis/build/piwik/plugin-CustomDimensions/piwik/plugins/CustomDimensions/tests/System/processed/test___API.getSegmentsMetadata.xml' because a new segment was added in core

    Example

this failing build https://travis-ci.org/piwik/plugin-CustomDimensions/jobs/106380038

Solution

the system test plugins/CustomDimensions/tests/System/processed/test___API.getSegmentsMetadata.xml should only test for those segments that are added by the plugin.

Outcome

the build for CustomDimensions will fail less often and make us more agile and move faster without breaking builds.

@tsteur commented on June 10th 2016 Member

As it is a system test it should kind of is supposed to fail and it's good when it's fail. That's the point of system tests. The test could be rewritten to an integration test but then we don't know if it actually works. I think it would be better to rather create better tools to fix system tests more easily

@mattab commented on July 8th 2016 Member

I think it would be better to rather create better tools to fix system tests more easily

Correct & wise thought :+1:

Powered by GitHub Issue Mirror