• ActionsDataTable_unflattened
• ViewDataTableTest_9_normal_table
• Overlay tests in build 5905
• Dashboard test, Bar Graph was not loaded (quite rare)
• Actions report truncation sometimes fail and truncates too much
• Transitions report displays values on the right, randomly build 6674

The Transitions random failure was fixed. Now we often see this error: http://builds-artifacts.piwik.org/ui-tests.master/10667.7/screenshot-diffs/singlediff.html?processed=../processed-ui-screenshots/ActionsDataTable_column_sorted.png&expected=ActionsDataTable_column_sorted.png&github=ActionsDataTable_column_sorted.png

but unfortunately we still don't know how to prevent this random failure. a hard to find edge case in Javascript or css or...

Moving to Short term because now random UI test failure is almost the only reason for failed CI builds. Needed for #6544

There is some feature in mocha to retry tests: https://www.npmjs.com/package/mocha-retry

we agree with what the devs says eg. Why you shouldn't use this? Tests that pass sometimes, and sometimes they don't are a bad smell. Your tests should be consistent: either they always pass, or they always fail. So, don't you this Mocha plugin, instead make your tests consistent.

however in this case i reckon none of us know how to fix the UI tests, so maybe a retry (only for those tests that are know to randomly fail) would be a correct solution

I asked the developer of mocha-retry for help in giggio/mocha-retry#6

If we can't make mocha-retry work maybe we could try this other tool: https://github.com/lennym/mocha-extra-shot

Answer from the dev of mocha-retry:

Yes, my version worked, but broke the browser, they said. I haven't updated it since then. They said they would look into it, but never did. Maybe you would like to help and see what went wrong there? I never got to it, and now I am on a project on a different technology, I will not find time to work on this for quite a while.

So it looks like we're back to square one, maybe the only way will be to actually solve the random screenshot rendering issues...

Here is a typical random screenshot failure: http://builds-artifacts.piwik.org/ui-tests.master/13257.7/screenshot-diffs/diffviewer.html

• for UIIntegrationTest_visitors_visitorlog and UIIntegrationTest_ecommerce_log the random failure is always the same and is for the "Goal count block":

Maybe we could change the positioning of this element, eg. position it relative to the right column, and this could prevent the random UI failure?

@mattab that's exactly what I was working on yesterday ;)

Here is the UI diff: http://builds-artifacts.piwik.org/ui-tests.fix-ui-tests-visitor-log/13240.7/screenshot-diffs/diffviewer.html What I did is disable the float and have 2 rows of icons. I can create a pull request later.

Here is the change for the visitor log: #8090

Note - in case we notice that we still have as many random build failures, we should revert this pull request: #8167

Moving to 2.14.1 to make an even stronger push for having Green builds, every day, for our UI screenshot tests. The team is all very much looking forward to reach this beautiful goal!

• It seems most of the UI random failures were fixed! this is more than great news for us. ☀️ ❤️
  • The Opt-out ui test is AFAIK the only one that failed (only once) in the last 10 builds or so http://builds-artifacts.piwik.org/ui-tests.master/13783.7/OptOutForm_safari-opted-out
• Build running long
  • The build takes between 38 and 46 minutes.
  • We are close from the 50min timeout and will soon have this issue
  • out of last 10 builds, 20% of the them timed out and took more than 50minutes.

Created separate issue: Screenshot test CI job is running too long #8222

Opt-out with safari agent, seems to be the last randomly failing UI test: http://builds-artifacts.piwik.org/ui-tests.8290/14030.7/

@diosmosis maybe you have some idea what could cause this UI test to randomly fail (it seems to work more than half of the time)

Along with "opt-out with safari use case", one of our newly added UI test for the Maps, randomly fail quite rarely. I've seen this failure 2-3 times, last one being: http://builds-artifacts.piwik.org/ui-tests.master/14138.7/ which was around midnight UTC. I guess these two tests fail when the build runs shortly after midnight UTC.

Edit: actually it's now 7AM UTC and still the build is failing https://travis-ci.org/piwik/piwik/jobs/70761848 so the issue is not only around midnight utc

We're really not sure how to fix any remaining random UI issue, but we will keep an eye on these issues.

It seems that all random issues are finally fixed!!! OMG Happiness is overwhelming me right now ;-) Well done for sticking with us while this was sorted out (took more than one year and dozens of little issues!)

Oh no I was mistaken, random failures still occur.... Maybe the only way going forward will be to use electron instead of phantomjs. We will investigate over next 6 months

In Piwik 3 we've upgraded Phantomjs to phantomjs 2 since it was required for compatibility with our heavier use of modern JavaScript (AngularJS), but we still have random screenshot fails. Now tracked in #10536