This is another one I've seen that doesn't load the css https://builds-artifacts.matomo.org/matomo-org/matomo/m-6032-export-transitions/49534/UIIntegrationTest_admin_themes.png

There is one that always failed on my pr.

  1) Overlay
       should load overlay correctly when coming from an widgetized action report:
          Cannot read property 'hover' of null

Just a thought, is that possible we only run tests on the parts we changed. like a subModule or Composer package :)

There is a general problem with automatically retrying UI tests. We have a lot tests that rely on the state of the previous test or that perform actions that can't be done again (like adding or removing stuff). So we could only rerun the whole test suite if one test failed, which then could cause timeouts if a long running test suite is rerun automatically.

Regarding the not loaded CSS/JS. I already added a lot code that automatically tries to load the CSS/JS again if the request fails, but there still seem to be some circumstances where all requests serve an empty response, so I guess the generated and cached CSS/JS is kind of empty and re-requesting it doesn't help.
I was never able to reproduce that locally, so this might be some travis specific error 🤷

@peterhashair I'll try to fix that failing Overlay UI test. Not yet sure why it fails, as it's working correctly locally.
And we are running all UI tests on purpose. Only running some specific ones might have the risk, that a change accidentally changed something else and you might not see that.

The overlay UI test should be fixed with #18084

Get this on the UI test, I guess there is a timeout or ssl on the test.

Note the premium feature UI test will be fixed by #18098

Another randomly failing one: https://builds-artifacts.matomo.org/matomo-org/matomo/m-3513-stick-table-top/50217/Dashboard_change_layout.png

Not sure that one is just me or everyone.

And this one.

It seems this starts to fail randomly. eg: https://app.travis-ci.com/github/matomo-org/matomo/jobs/547051754

Found this one today, apparently random

There was 1 failure:

1) Piwik\Plugins\DeviceDetectorCache\tests\Integration\WarmDeviceDetectorCacheTest::testDoesClearExistingFilesFromCacheByDefaultWhenTooManyEntriesExist

Failed asserting that file "/home/travis/build/matomo-org/matomo/tmp/devicecachetests/6d4/6d44eab61dcf5b1f0e6fa8f6595ee1bc.php" exists.

/home/travis/build/matomo-org/matomo/plugins/DeviceDetectorCache/tests/Integration/WarmDeviceDetectorCacheTest.php:132

I guess this is old one.

1) Piwik\Plugins\CoreConsole\tests\System\ArchiveCronTest::testArchivePhpCron
Piwik\Plugins\CoreConsole\tests\System\ArchiveCronTest: Differences with expected in '/home/travis/build/matomo-org/matomo/plugins/CoreConsole/tests/System/processed/test_ArchiveCronTest_segmentForToday_noOptions__VisitsSummary.get_day.xml'
Failed asserting that two DOM documents are equal.
--- Expected
+++ Actual
@@ @@
   <result idSite="1">
     <nb_uniq_visitors>1</nb_uniq_visitors>
     <nb_users>0</nb_users>
-    <nb_visits>2</nb_visits>
+    <nb_visits>1</nb_visits>
     <nb_actions>2</nb_actions>
-    <nb_visits_converted>2</nb_visits_converted>
-    <bounce_count>2</bounce_count>
-    <sum_visit_length>0</sum_visit_length>
-    <max_actions>1</max_actions>
-    <bounce_rate>100%</bounce_rate>
-    <nb_actions_per_visit>1</nb_actions_per_visit>
-    <avg_time_on_site>0</avg_time_on_site>
+    <nb_visits_converted>1</nb_visits_converted>
+    <bounce_count>0</bounce_count>
+    <sum_visit_length>1</sum_visit_length>
+    <max_actions>2</max_actions>
+    <bounce_rate>0%</bounce_rate>
+    <nb_actions_per_visit>2</nb_actions_per_visit>
+    <avg_time_on_site>1</avg_time_on_site>
   </result>
   <result idSite="2"/>
   <result idSite="3"/>

There is a new one.

1) TwoFactorAuth
     should be possible to disable two factor step 3 verified:
        waiting for selector `.userSettings2FA` failed: timeout 30000ms exceeded
          Url to reproduce: http://localhost/tests/PHPUnit/proxy/index.php?module=TwoFactorAuth&action=disableTwoFactorAuth&idSite=1&period=day&date=2018-03-04&disableNonce=fba9cf662527b39fc9b1b662128d59ff&updated=1
          Screenshot of failure: /home/travis/build/matomo-org/matomo/tests/UI/processed-ui-screenshots/should_be_possible_to_disable_two_factor_step_3_verified_failure.png


     Rendering logs:
     
     ```

Not sure this is random. After ce27828 start to fail

1) Piwik\Plugins\UserId\tests\System\ApiTest::testApi with data set #1 ('UserId.getUsers', array('2010-02-01,2011-02-01', array('range'), 1, ''))
Piwik\Plugins\UserId\tests\System\ApiTest: Differences with expected in '/home/travis/build/matomo-org/matomo/plugins/UserId/tests/System/processed/test___UserId.getUsers_range.xml'
Failed asserting that two DOM documents are equal.
--- Expected
+++ Actual
@@ @@
    <label>user3</label>
    <nb_visits>31</nb_visits>
    <nb_actions>44</nb_actions>
-    <max_actions>4</max_actions>
-    <sum_visit_length>11</sum_visit_length>
-    <bounce_count>20</bounce_count>
+    <max_actions>3</max_actions>
+    <sum_visit_length>12</sum_visit_length>
+    <bounce_count>19</bounce_count>
    <nb_visits_converted>0</nb_visits_converted>
    <sum_daily_nb_uniq_visitors>30</sum_daily_nb_uniq_visitors>
    <sum_daily_nb_users>30</sum_daily_nb_users>
@@ @@
    <label>user2</label>
    <nb_visits>21</nb_visits>
    <nb_actions>29</nb_actions>
-    <max_actions>3</max_actions>
-    <sum_visit_length>7</sum_visit_length>
-    <bounce_count>14</bounce_count>
+    <max_actions>2</max_actions>
+    <sum_visit_length>8</sum_visit_length>
+    <bounce_count>13</bounce_count>
    <nb_visits_converted>0</nb_visits_converted>
    <sum_daily_nb_uniq_visitors>20</sum_daily_nb_uniq_visitors>
    <sum_daily_nb_users>20</sum_daily_nb_users>

Not sure this is random. After ce27828 start to fail

1) Piwik\Plugins\UserId\tests\System\ApiTest::testApi with data set #1 ('UserId.getUsers', array('2010-02-01,2011-02-01', array('range'), 1, ''))
Piwik\Plugins\UserId\tests\System\ApiTest: Differences with expected in '/home/travis/build/matomo-org/matomo/plugins/UserId/tests/System/processed/test___UserId.getUsers_range.xml'
Failed asserting that two DOM documents are equal.
--- Expected
+++ Actual
@@ @@
    <label>user3</label>
    <nb_visits>31</nb_visits>
    <nb_actions>44</nb_actions>
-    <max_actions>4</max_actions>
-    <sum_visit_length>11</sum_visit_length>
-    <bounce_count>20</bounce_count>
+    <max_actions>3</max_actions>
+    <sum_visit_length>12</sum_visit_length>
+    <bounce_count>19</bounce_count>
    <nb_visits_converted>0</nb_visits_converted>
    <sum_daily_nb_uniq_visitors>30</sum_daily_nb_uniq_visitors>
    <sum_daily_nb_users>30</sum_daily_nb_users>
@@ @@
    <label>user2</label>
    <nb_visits>21</nb_visits>
    <nb_actions>29</nb_actions>
-    <max_actions>3</max_actions>
-    <sum_visit_length>7</sum_visit_length>
-    <bounce_count>14</bounce_count>
+    <max_actions>2</max_actions>
+    <sum_visit_length>8</sum_visit_length>
+    <bounce_count>13</bounce_count>
    <nb_visits_converted>0</nb_visits_converted>
    <sum_daily_nb_uniq_visitors>20</sum_daily_nb_uniq_visitors>
    <sum_daily_nb_users>20</sum_daily_nb_users>

@peterhashair Same is happening with me on #18732

@AltamashShaikh it seems like started to fail quite often, will have a deep look now.

It seems like related to travis. github action works https://github.com/matomo-org/matomo/actions/runs/1793294031

Not sure this is random. After ce27828 start to fail

1) Piwik\Plugins\UserId\tests\System\ApiTest::testApi with data set #1 ('UserId.getUsers', array('2010-02-01,2011-02-01', array('range'), 1, ''))
Piwik\Plugins\UserId\tests\System\ApiTest: Differences with expected in '/home/travis/build/matomo-org/matomo/plugins/UserId/tests/System/processed/test___UserId.getUsers_range.xml'
Failed asserting that two DOM documents are equal.
--- Expected
+++ Actual
@@ @@
    <label>user3</label>
    <nb_visits>31</nb_visits>
    <nb_actions>44</nb_actions>
-    <max_actions>4</max_actions>
-    <sum_visit_length>11</sum_visit_length>
-    <bounce_count>20</bounce_count>
+    <max_actions>3</max_actions>
+    <sum_visit_length>12</sum_visit_length>
+    <bounce_count>19</bounce_count>
    <nb_visits_converted>0</nb_visits_converted>
    <sum_daily_nb_uniq_visitors>30</sum_daily_nb_uniq_visitors>
    <sum_daily_nb_users>30</sum_daily_nb_users>
@@ @@
    <label>user2</label>
    <nb_visits>21</nb_visits>
    <nb_actions>29</nb_actions>
-    <max_actions>3</max_actions>
-    <sum_visit_length>7</sum_visit_length>
-    <bounce_count>14</bounce_count>
+    <max_actions>2</max_actions>
+    <sum_visit_length>8</sum_visit_length>
+    <bounce_count>13</bounce_count>
    <nb_visits_converted>0</nb_visits_converted>
    <sum_daily_nb_uniq_visitors>20</sum_daily_nb_uniq_visitors>
    <sum_daily_nb_users>20</sum_daily_nb_users>

Seems this fail randomly very often rencetly

1) Piwik\Plugins\CoreConsole\tests\System\ArchiveCronTest::testArchivePhpCron

I'm actually tempted to close this issue simply. We should always try to keep our builds green. If someone sees that a test fails randomly quite often they can simply try to improve that behavior and create a PR or create an issue instead that mentions that specific test only. Gathering more and more tests that might have failed randomly in one issue makes it extremely hard to even work on it. Some of tests here were meanwhile fixed, others might start to fail while someone might work on that issue and new comments with failing tests might be added.
And in general to postpone fixing some (randomly) failing tests is never a good idea as it can meanwhile always produce regressions making more and more users unhappy.

Cool I will close this now and we can create individual issues for any tests that fail in future.