Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AllTests CI job: a test is failing but the build job is green #11695

Closed
mattab opened this issue May 12, 2017 · 1 comment
Closed

AllTests CI job: a test is failing but the build job is green #11695

mattab opened this issue May 12, 2017 · 1 comment
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Milestone
3.1.0

Comments

@mattab
Copy link
Member

mattab commented May 12, 2017

Example #1

https://travis-ci.org/piwik/piwik/jobs/231488610

we see there was a failure in the system tests:

There was 1 failure:

1) Piwik\Tests\System\CsvExportTest::testApi with data set #3 ('Live.getLastVisitsDetails', array(1, '2010-01-03 11:22:33', 'csv', array('serverDate,actionDetails,serv...Action')))

Piwik\Tests\System\CsvExportTest: Differences with expected in '/home/travis/build/piwik/piwik/tests/PHPUnit/Framework/TestCase/../../System/processed/test_csvExport__Live.getLastVisitsDetails_day.csv'

Expected

When a test fails in the AllTests job, the job should fail

Got instead

The build is green, maybe because the UnitTests are passing?

Example #2

in https://travis-ci.org/piwik/piwik/jobs/231488611 one of the integration test failed:

1) Piwik\Tests\Integration\ArchiveTest::testExistingArchivesAreReplaced

Failed asserting that 1 matches expected 2.`

The PluginTests are passing.

Got

the job was green

Expected

the job to be red/failing as one test failed
@mattab mattab added the c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. label May 12, 2017
@mattab mattab added this to the 3.0.5 milestone May 12, 2017
@mattab mattab changed the title AllTests CI job: a test is failing the but the build job is green AllTests CI job: a test is failing but the build job is green May 18, 2017
@mneudert mneudert mentioned this issue Jun 24, 2017
Merged
@sgiehl
Copy link
Member

sgiehl commented Jun 24, 2017

fixed in matomo-org/travis-scripts#34

@sgiehl sgiehl closed this as completed Jun 24, 2017
@mattab mattab added the not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org. label Sep 11, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. not-in-changelog For issues or pull requests that should not be included in our release changelog on matomo.org.
Projects
None yet
Milestone
3.1.0
Development

No branches or pull requests
2 participants
@mattab @sgiehl