@tsteur opened this Issue on September 30th 2021 Member

These ones fail randomly quite often AFAIK

Could we maybe retry them when they fail? or try to fix them in a different way?

@bx80 commented on September 30th 2021 Contributor
@peterhashair commented on September 30th 2021 Contributor

There is one that always failed on my pr.

  1) Overlay
       should load overlay correctly when coming from an widgetized action report:
          Cannot read property 'hover' of null

Just a thought, is that possible we only run tests on the parts we changed. like a subModule or Composer package :)

@sgiehl commented on September 30th 2021 Member

There is a general problem with automatically retrying UI tests. We have a lot tests that rely on the state of the previous test or that perform actions that can't be done again (like adding or removing stuff). So we could only rerun the whole test suite if one test failed, which then could cause timeouts if a long running test suite is rerun automatically.

Regarding the not loaded CSS/JS. I already added a lot code that automatically tries to load the CSS/JS again if the request fails, but there still seem to be some circumstances where all requests serve an empty response, so I guess the generated and cached CSS/JS is kind of empty and re-requesting it doesn't help.
I was never able to reproduce that locally, so this might be some travis specific error 🤷

@peterhashair I'll try to fix that failing Overlay UI test. Not yet sure why it fails, as it's working correctly locally.
And we are running all UI tests on purpose. Only running some specific ones might have the risk, that a change accidentally changed something else and you might not see that.

@sgiehl commented on September 30th 2021 Member

The overlay UI test should be fixed with https://github.com/matomo-org/matomo/pull/18084

@peterhashair commented on October 5th 2021 Contributor

Get this on the UI test, I guess there is a timeout or ssl on the test.

@tsteur commented on October 7th 2021 Member

Note the premium feature UI test will be fixed by https://github.com/matomo-org/matomo/issues/18098

@justinvelluppillai commented on October 19th 2021 Contributor
@peterhashair commented on October 20th 2021 Contributor

Not sure that one is just me or everyone.

@peterhashair commented on October 20th 2021 Contributor

And this one.

@peterhashair commented on November 8th 2021 Contributor

It seems this starts to fail randomly. eg: https://app.travis-ci.com/github/matomo-org/matomo/jobs/547051754

@justinvelluppillai commented on December 15th 2021 Contributor

Found this one today, apparently random

There was 1 failure:

1) Piwik\Plugins\DeviceDetectorCache\tests\Integration\WarmDeviceDetectorCacheTest::testDoesClearExistingFilesFromCacheByDefaultWhenTooManyEntriesExist

Failed asserting that file "/home/travis/build/matomo-org/matomo/tmp/devicecachetests/6d4/6d44eab61dcf5b1f0e6fa8f6595ee1bc.php" exists.

@peterhashair commented on January 11th 2022 Contributor

I guess this is old one.

1) Piwik\Plugins\CoreConsole\tests\System\ArchiveCronTest::testArchivePhpCron
Piwik\Plugins\CoreConsole\tests\System\ArchiveCronTest: Differences with expected in '/home/travis/build/matomo-org/matomo/plugins/CoreConsole/tests/System/processed/test_ArchiveCronTest_segmentForToday_noOptions__VisitsSummary.get_day.xml'
Failed asserting that two DOM documents are equal.
--- Expected
+++ Actual
@@ @@
   <result idSite="1">
-    <nb_visits>2</nb_visits>
+    <nb_visits>1</nb_visits>
-    <nb_visits_converted>2</nb_visits_converted>
-    <bounce_count>2</bounce_count>
-    <sum_visit_length>0</sum_visit_length>
-    <max_actions>1</max_actions>
-    <bounce_rate>100%</bounce_rate>
-    <nb_actions_per_visit>1</nb_actions_per_visit>
-    <avg_time_on_site>0</avg_time_on_site>
+    <nb_visits_converted>1</nb_visits_converted>
+    <bounce_count>0</bounce_count>
+    <sum_visit_length>1</sum_visit_length>
+    <max_actions>2</max_actions>
+    <bounce_rate>0%</bounce_rate>
+    <nb_actions_per_visit>2</nb_actions_per_visit>
+    <avg_time_on_site>1</avg_time_on_site>
   <result idSite="2"/>
   <result idSite="3"/>
@peterhashair commented on January 11th 2022 Contributor

There is a new one.

  1) TwoFactorAuth
       should be possible to disable two factor step 3 verified:
          waiting for selector `.userSettings2FA` failed: timeout 30000ms exceeded
            Url to reproduce: http://localhost/tests/PHPUnit/proxy/index.php?module=TwoFactorAuth&action=disableTwoFactorAuth&idSite=1&period=day&date=2018-03-04&disableNonce=fba9cf662527b39fc9b1b662128d59ff&updated=1
            Screenshot of failure: /home/travis/build/matomo-org/matomo/tests/UI/processed-ui-screenshots/should_be_possible_to_disable_two_factor_step_3_verified_failure.png

       Rendering logs:
Powered by GitHub Issue Mirror