@tsteur opened this Issue on September 30th 2021 Member

These ones fail randomly quite often AFAIK

Could we maybe retry them when they fail? or try to fix them in a different way?

@bx80 commented on September 30th 2021 Contributor
@peterhashair commented on September 30th 2021 Contributor

There is one that always failed on my pr.

  1) Overlay
       should load overlay correctly when coming from an widgetized action report:
          Cannot read property 'hover' of null

Just a thought, is that possible we only run tests on the parts we changed. like a subModule or Composer package :)

@sgiehl commented on September 30th 2021 Member

There is a general problem with automatically retrying UI tests. We have a lot tests that rely on the state of the previous test or that perform actions that can't be done again (like adding or removing stuff). So we could only rerun the whole test suite if one test failed, which then could cause timeouts if a long running test suite is rerun automatically.

Regarding the not loaded CSS/JS. I already added a lot code that automatically tries to load the CSS/JS again if the request fails, but there still seem to be some circumstances where all requests serve an empty response, so I guess the generated and cached CSS/JS is kind of empty and re-requesting it doesn't help.
I was never able to reproduce that locally, so this might be some travis specific error 🤷

@peterhashair I'll try to fix that failing Overlay UI test. Not yet sure why it fails, as it's working correctly locally.
And we are running all UI tests on purpose. Only running some specific ones might have the risk, that a change accidentally changed something else and you might not see that.

@sgiehl commented on September 30th 2021 Member

The overlay UI test should be fixed with https://github.com/matomo-org/matomo/pull/18084

@peterhashair commented on October 5th 2021 Contributor

Get this on the UI test, I guess there is a timeout or ssl on the test.

@tsteur commented on October 7th 2021 Member

Note the premium feature UI test will be fixed by https://github.com/matomo-org/matomo/issues/18098

Powered by GitHub Issue Mirror