These ones fail randomly quite often AFAIK
Could we maybe retry them when they fail? or try to fix them in a different way?
This is another one I've seen that doesn't load the css https://builds-artifacts.matomo.org/matomo-org/matomo/m-6032-export-transitions/49534/UIIntegrationTest_admin_themes.png
There is one that always failed on my pr.
1) Overlay should load overlay correctly when coming from an widgetized action report: Cannot read property 'hover' of null
Just a thought, is that possible we only run tests on the parts we changed. like a subModule or Composer package :)
There is a general problem with automatically retrying UI tests. We have a lot tests that rely on the state of the previous test or that perform actions that can't be done again (like adding or removing stuff). So we could only rerun the whole test suite if one test failed, which then could cause timeouts if a long running test suite is rerun automatically.
Regarding the not loaded CSS/JS. I already added a lot code that automatically tries to load the CSS/JS again if the request fails, but there still seem to be some circumstances where all requests serve an empty response, so I guess the generated and cached CSS/JS is kind of empty and re-requesting it doesn't help.
I was never able to reproduce that locally, so this might be some travis specific error 🤷
@peterhashair I'll try to fix that failing Overlay UI test. Not yet sure why it fails, as it's working correctly locally.
And we are running all UI tests on purpose. Only running some specific ones might have the risk, that a change accidentally changed something else and you might not see that.
The overlay UI test should be fixed with https://github.com/matomo-org/matomo/pull/18084
Get this on the UI test, I guess there is a timeout or ssl on the test.
Note the premium feature UI test will be fixed by https://github.com/matomo-org/matomo/issues/18098
Another randomly failing one: https://builds-artifacts.matomo.org/matomo-org/matomo/m-3513-stick-table-top/50217/Dashboard_change_layout.png
Not sure that one is just me or everyone.
And this one.
It seems this starts to fail randomly. eg: https://app.travis-ci.com/github/matomo-org/matomo/jobs/547051754
Found this one today, apparently random
There was 1 failure: 1) Piwik\Plugins\DeviceDetectorCache\tests\Integration\WarmDeviceDetectorCacheTest::testDoesClearExistingFilesFromCacheByDefaultWhenTooManyEntriesExist Failed asserting that file "/home/travis/build/matomo-org/matomo/tmp/devicecachetests/6d4/6d44eab61dcf5b1f0e6fa8f6595ee1bc.php" exists. /home/travis/build/matomo-org/matomo/plugins/DeviceDetectorCache/tests/Integration/WarmDeviceDetectorCacheTest.php:132
I guess this is old one.
1) Piwik\Plugins\CoreConsole\tests\System\ArchiveCronTest::testArchivePhpCron Piwik\Plugins\CoreConsole\tests\System\ArchiveCronTest: Differences with expected in '/home/travis/build/matomo-org/matomo/plugins/CoreConsole/tests/System/processed/test_ArchiveCronTest_segmentForToday_noOptions__VisitsSummary.get_day.xml' Failed asserting that two DOM documents are equal. --- Expected +++ Actual @@ @@ <result idSite="1"> <nb_uniq_visitors>1</nb_uniq_visitors> <nb_users>0</nb_users> - <nb_visits>2</nb_visits> + <nb_visits>1</nb_visits> <nb_actions>2</nb_actions> - <nb_visits_converted>2</nb_visits_converted> - <bounce_count>2</bounce_count> - <sum_visit_length>0</sum_visit_length> - <max_actions>1</max_actions> - <bounce_rate>100%</bounce_rate> - <nb_actions_per_visit>1</nb_actions_per_visit> - <avg_time_on_site>0</avg_time_on_site> + <nb_visits_converted>1</nb_visits_converted> + <bounce_count>0</bounce_count> + <sum_visit_length>1</sum_visit_length> + <max_actions>2</max_actions> + <bounce_rate>0%</bounce_rate> + <nb_actions_per_visit>2</nb_actions_per_visit> + <avg_time_on_site>1</avg_time_on_site> </result> <result idSite="2"/> <result idSite="3"/>
There is a new one.
1) TwoFactorAuth should be possible to disable two factor step 3 verified: waiting for selector `.userSettings2FA` failed: timeout 30000ms exceeded Url to reproduce: http://localhost/tests/PHPUnit/proxy/index.php?module=TwoFactorAuth&action=disableTwoFactorAuth&idSite=1&period=day&date=2018-03-04&disableNonce=fba9cf662527b39fc9b1b662128d59ff&updated=1 Screenshot of failure: /home/travis/build/matomo-org/matomo/tests/UI/processed-ui-screenshots/should_be_possible_to_disable_two_factor_step_3_verified_failure.png Rendering logs: