@tsteur opened this Issue on September 30th 2021 Member

These ones fail randomly quite often AFAIK

Could we maybe retry them when they fail? or try to fix them in a different way?

@bx80 commented on September 30th 2021 Contributor
@peterhashair commented on September 30th 2021 Contributor

There is one that always failed on my pr.

  1) Overlay
       should load overlay correctly when coming from an widgetized action report:
          Cannot read property 'hover' of null

Just a thought, is that possible we only run tests on the parts we changed. like a subModule or Composer package :)

@sgiehl commented on September 30th 2021 Member

There is a general problem with automatically retrying UI tests. We have a lot tests that rely on the state of the previous test or that perform actions that can't be done again (like adding or removing stuff). So we could only rerun the whole test suite if one test failed, which then could cause timeouts if a long running test suite is rerun automatically.

Regarding the not loaded CSS/JS. I already added a lot code that automatically tries to load the CSS/JS again if the request fails, but there still seem to be some circumstances where all requests serve an empty response, so I guess the generated and cached CSS/JS is kind of empty and re-requesting it doesn't help.
I was never able to reproduce that locally, so this might be some travis specific error 🤷

@peterhashair I'll try to fix that failing Overlay UI test. Not yet sure why it fails, as it's working correctly locally.
And we are running all UI tests on purpose. Only running some specific ones might have the risk, that a change accidentally changed something else and you might not see that.

@sgiehl commented on September 30th 2021 Member

The overlay UI test should be fixed with https://github.com/matomo-org/matomo/pull/18084

@peterhashair commented on October 5th 2021 Contributor

Get this on the UI test, I guess there is a timeout or ssl on the test.
image

@tsteur commented on October 7th 2021 Member

Note the premium feature UI test will be fixed by https://github.com/matomo-org/matomo/issues/18098

@justinvelluppillai commented on October 19th 2021 Contributor
@peterhashair commented on October 20th 2021 Contributor

Not sure that one is just me or everyone.
image

@peterhashair commented on October 20th 2021 Contributor

And this one.
image

@peterhashair commented on November 8th 2021 Contributor

It seems this starts to fail randomly. eg: https://app.travis-ci.com/github/matomo-org/matomo/jobs/547051754
image

@justinvelluppillai commented on December 15th 2021 Contributor

Found this one today, apparently random

There was 1 failure:

1) Piwik\Plugins\DeviceDetectorCache\tests\Integration\WarmDeviceDetectorCacheTest::testDoesClearExistingFilesFromCacheByDefaultWhenTooManyEntriesExist

Failed asserting that file "/home/travis/build/matomo-org/matomo/tmp/devicecachetests/6d4/6d44eab61dcf5b1f0e6fa8f6595ee1bc.php" exists.

/home/travis/build/matomo-org/matomo/plugins/DeviceDetectorCache/tests/Integration/WarmDeviceDetectorCacheTest.php:132
@peterhashair commented on January 11th 2022 Contributor

I guess this is old one.

1) Piwik\Plugins\CoreConsole\tests\System\ArchiveCronTest::testArchivePhpCron
Piwik\Plugins\CoreConsole\tests\System\ArchiveCronTest: Differences with expected in '/home/travis/build/matomo-org/matomo/plugins/CoreConsole/tests/System/processed/test_ArchiveCronTest_segmentForToday_noOptions__VisitsSummary.get_day.xml'
Failed asserting that two DOM documents are equal.
--- Expected
+++ Actual
@@ @@
   <result idSite="1">
     <nb_uniq_visitors>1</nb_uniq_visitors>
     <nb_users>0</nb_users>
-    <nb_visits>2</nb_visits>
+    <nb_visits>1</nb_visits>
     <nb_actions>2</nb_actions>
-    <nb_visits_converted>2</nb_visits_converted>
-    <bounce_count>2</bounce_count>
-    <sum_visit_length>0</sum_visit_length>
-    <max_actions>1</max_actions>
-    <bounce_rate>100%</bounce_rate>
-    <nb_actions_per_visit>1</nb_actions_per_visit>
-    <avg_time_on_site>0</avg_time_on_site>
+    <nb_visits_converted>1</nb_visits_converted>
+    <bounce_count>0</bounce_count>
+    <sum_visit_length>1</sum_visit_length>
+    <max_actions>2</max_actions>
+    <bounce_rate>0%</bounce_rate>
+    <nb_actions_per_visit>2</nb_actions_per_visit>
+    <avg_time_on_site>1</avg_time_on_site>
   </result>
   <result idSite="2"/>
   <result idSite="3"/>
@peterhashair commented on January 11th 2022 Contributor

There is a new one.

  1) TwoFactorAuth
       should be possible to disable two factor step 3 verified:
          waiting for selector `.userSettings2FA` failed: timeout 30000ms exceeded
            Url to reproduce: http://localhost/tests/PHPUnit/proxy/index.php?module=TwoFactorAuth&action=disableTwoFactorAuth&idSite=1&period=day&date=2018-03-04&disableNonce=fba9cf662527b39fc9b1b662128d59ff&updated=1
            Screenshot of failure: /home/travis/build/matomo-org/matomo/tests/UI/processed-ui-screenshots/should_be_possible_to_disable_two_factor_step_3_verified_failure.png

       Rendering logs:
@peterhashair commented on February 3rd 2022 Contributor

Not sure this is random. After ce2782827667e4cbf5162cfc7a84bfe657de96d0 start to fail

1) Piwik\Plugins\UserId\tests\System\ApiTest::testApi with data set <a href='/1'>#1</a> ('UserId.getUsers', array('2010-02-01,2011-02-01', array('range'), 1, ''))
Piwik\Plugins\UserId\tests\System\ApiTest: Differences with expected in '/home/travis/build/matomo-org/matomo/plugins/UserId/tests/System/processed/test___UserId.getUsers_range.xml'
Failed asserting that two DOM documents are equal.
--- Expected
+++ Actual
@@ @@
     <label>user3</label>
     <nb_visits>31</nb_visits>
     <nb_actions>44</nb_actions>
-    <max_actions>4</max_actions>
-    <sum_visit_length>11</sum_visit_length>
-    <bounce_count>20</bounce_count>
+    <max_actions>3</max_actions>
+    <sum_visit_length>12</sum_visit_length>
+    <bounce_count>19</bounce_count>
     <nb_visits_converted>0</nb_visits_converted>
     <sum_daily_nb_uniq_visitors>30</sum_daily_nb_uniq_visitors>
     <sum_daily_nb_users>30</sum_daily_nb_users>
@@ @@
     <label>user2</label>
     <nb_visits>21</nb_visits>
     <nb_actions>29</nb_actions>
-    <max_actions>3</max_actions>
-    <sum_visit_length>7</sum_visit_length>
-    <bounce_count>14</bounce_count>
+    <max_actions>2</max_actions>
+    <sum_visit_length>8</sum_visit_length>
+    <bounce_count>13</bounce_count>
     <nb_visits_converted>0</nb_visits_converted>
     <sum_daily_nb_uniq_visitors>20</sum_daily_nb_uniq_visitors>
     <sum_daily_nb_users>20</sum_daily_nb_users>
@AltamashShaikh commented on February 4th 2022 Contributor

Not sure this is random. After ce27828 start to fail

1) Piwik\Plugins\UserId\tests\System\ApiTest::testApi with data set <a href='/1'>#1</a> ('UserId.getUsers', array('2010-02-01,2011-02-01', array('range'), 1, ''))
Piwik\Plugins\UserId\tests\System\ApiTest: Differences with expected in '/home/travis/build/matomo-org/matomo/plugins/UserId/tests/System/processed/test___UserId.getUsers_range.xml'
Failed asserting that two DOM documents are equal.
--- Expected
+++ Actual
@@ @@
    <label>user3</label>
    <nb_visits>31</nb_visits>
    <nb_actions>44</nb_actions>
-    <max_actions>4</max_actions>
-    <sum_visit_length>11</sum_visit_length>
-    <bounce_count>20</bounce_count>
+    <max_actions>3</max_actions>
+    <sum_visit_length>12</sum_visit_length>
+    <bounce_count>19</bounce_count>
    <nb_visits_converted>0</nb_visits_converted>
    <sum_daily_nb_uniq_visitors>30</sum_daily_nb_uniq_visitors>
    <sum_daily_nb_users>30</sum_daily_nb_users>
@@ @@
    <label>user2</label>
    <nb_visits>21</nb_visits>
    <nb_actions>29</nb_actions>
-    <max_actions>3</max_actions>
-    <sum_visit_length>7</sum_visit_length>
-    <bounce_count>14</bounce_count>
+    <max_actions>2</max_actions>
+    <sum_visit_length>8</sum_visit_length>
+    <bounce_count>13</bounce_count>
    <nb_visits_converted>0</nb_visits_converted>
    <sum_daily_nb_uniq_visitors>20</sum_daily_nb_uniq_visitors>
    <sum_daily_nb_users>20</sum_daily_nb_users>

@peterhashair Same is happening with me on #18732

@peterhashair commented on February 4th 2022 Contributor

@AltamashShaikh it seems like started to fail quite often, will have a deep look now.

@peterhashair commented on February 4th 2022 Contributor

It seems like related to travis. github action works https://github.com/matomo-org/matomo/actions/runs/1793294031

Not sure this is random. After ce27828 start to fail

1) Piwik\Plugins\UserId\tests\System\ApiTest::testApi with data set <a href='/1'>#1</a> ('UserId.getUsers', array('2010-02-01,2011-02-01', array('range'), 1, ''))
Piwik\Plugins\UserId\tests\System\ApiTest: Differences with expected in '/home/travis/build/matomo-org/matomo/plugins/UserId/tests/System/processed/test___UserId.getUsers_range.xml'
Failed asserting that two DOM documents are equal.
--- Expected
+++ Actual
@@ @@
    <label>user3</label>
    <nb_visits>31</nb_visits>
    <nb_actions>44</nb_actions>
-    <max_actions>4</max_actions>
-    <sum_visit_length>11</sum_visit_length>
-    <bounce_count>20</bounce_count>
+    <max_actions>3</max_actions>
+    <sum_visit_length>12</sum_visit_length>
+    <bounce_count>19</bounce_count>
    <nb_visits_converted>0</nb_visits_converted>
    <sum_daily_nb_uniq_visitors>30</sum_daily_nb_uniq_visitors>
    <sum_daily_nb_users>30</sum_daily_nb_users>
@@ @@
    <label>user2</label>
    <nb_visits>21</nb_visits>
    <nb_actions>29</nb_actions>
-    <max_actions>3</max_actions>
-    <sum_visit_length>7</sum_visit_length>
-    <bounce_count>14</bounce_count>
+    <max_actions>2</max_actions>
+    <sum_visit_length>8</sum_visit_length>
+    <bounce_count>13</bounce_count>
    <nb_visits_converted>0</nb_visits_converted>
    <sum_daily_nb_uniq_visitors>20</sum_daily_nb_uniq_visitors>
    <sum_daily_nb_users>20</sum_daily_nb_users>
@peterhashair commented on February 16th 2022 Contributor

Seems this fail randomly very often rencetly

1) Piwik\Plugins\CoreConsole\tests\System\ArchiveCronTest::testArchivePhpCron
@sgiehl commented on April 4th 2022 Member

I'm actually tempted to close this issue simply. We should always try to keep our builds green. If someone sees that a test fails randomly quite often they can simply try to improve that behavior and create a PR or create an issue instead that mentions that specific test only. Gathering more and more tests that might have failed randomly in one issue makes it extremely hard to even work on it. Some of tests here were meanwhile fixed, others might start to fail while someone might work on that issue and new comments with failing tests might be added.
And in general to postpone fixing some (randomly) failing tests is never a good idea as it can meanwhile always produce regressions making more and more users unhappy.

@justinvelluppillai commented on April 4th 2022 Contributor

Cool I will close this now and we can create individual issues for any tests that fail in future.

This Issue was closed on April 4th 2022
Powered by GitHub Issue Mirror