Before investing any time in this we should check what is slow when running the UI tests. Is it server time? Is it network, JavaScript execution, CSS, ...? Maybe the application/library is only waiting for some resources to load or ... A first step could be to use remote debugging which is supported by Chrome, Firefox, Safari, to see where the time is actually spent. Maybe also worth profiling some tests/lib/screenshot-testing code.

Would be nice if PhantomJS could be still used so that we can use https://github.com/Huddle/PhantomCSS or similar at some point which is documented and has many more features etc

I just reduced defaultWaitTime here https://github.com/piwik/piwik/blob/master/tests/lib/screenshot-testing/support/page-renderer.js#L22 from 1 second to 100ms and it reduced the execution time from 15minutes to 7minutes on AWS. Unfortunately, also some tests fail now but it shows there seems to be potential to improve performance? Screenshots were taken about 3-4 times faster. I couldn't really understand what the code does so it might be nonsense.

I also noticed there are often wait times specified for instance along a click. See for example here 10 seconds wait time: https://github.com/piwik/piwik-ui-tests/blob/master/specs/Dashboard_spec.js#L182

If I understand it correct it waits in this case for 10 seconds before it continues to proceed? Maybe we can rather use something like "waitForCssSelectorPresent('.cssSelectorThatIsThereOnceLoaded')" to avoid such wait times? We used this a lot when writing selenium tests but it is more effort to write and maintain tests this way...

Edit: If I understand correct for each click there is a wait time of 1 second at least (there are > 170 clicks), sometimes wait time is up to 10seconds. Also sometimes there is a .wait(3000) which waits for 3 seconds. I think we wouldn't improve performance much with another engine

A similar solution is something like "spin" in Behat: remove all waits and retry each step until it works (with a 5-10 seconds timeout).

The timeouts are wait times before invoking the next action. This is mostly for waiting for animations I think. Including a way to check for those would mean you wouldn't need to wait for 1s. The 3s timeouts should be specifically for animations (so the screenshots don't look half faded in or whatever). The 10s timeouts are when we (me or matt) gave up on figuring out why the wait was needed.

Having wait times is to avoid "spin"-ing which would incur constant cpu use for something unimportant.

Would sometimes methods like those help? http://docs.casperjs.org/en/latest/modules/casper.html#waitforselector or http://docs.casperjs.org/en/latest/modules/casper.html#waituntilvisible

That's what we used to do in selenium tests in other projects and worked fine but there we didn't take screenshots. Or maybe we can check like every 50ms whether something is finished / done to avoid cpu usage. Anyway, this seems to be the slow part of the tests and probably not phantomjs. I was only wondering why they are so slow yesterday and I do understand it now

I don't think waitforselector will work since the element will be there, just animated. I don't know if being partially visible is considered visible for waituntilvisible.

Or maybe we can check like every 50ms whether something is finished / done to avoid cpu usage.

This is what is done now for AJAX requests, image loading & whatever else I could find a way to do it for. I couldn't figure out how to do it for jquery animations.

Note: I use LightTable sometimes which runs on node-webkit. The entire app is written in clojurescript, so it loads the entire app, the clojurescript compiler, every plugin's JS. Additionally w/ my plugin it loads angularjs + jquery + some other node extensions. And it does this in < 5s. Whereas phantomjs takes ~20-30s to load a Piwik page. Maybe Piwik's JS is ridiculously slow (or it's all the AJAX when loading a page), but there's definitely something amiss.

Could also try w/ https://github.com/atom/atom-shell (maintained by github).

We've migrated to Phantomjs2 for now

PhantomJS works well now. Can maybe reopen later if it becomes an issue again