At some point we should use Phantomjs 2.0 for our screenshot tests. If we are lucky (!) it may help us resolve some issues such as: #7147 #8034. Maybe it would also let us re-enable some of our disabled screenshot tests in #8142
Current blocker: http://phantomjs.org/download.html
Binary packages for Linux are still being prepared. There are still issues to be solved until a static build is available (see issue 12948 for more details).
If we didn't want to wait for official linux builds we could maybe use these PPA eg. https://launchpad.net/~tanguy-patte/+archive/ubuntu/phantomjs
I just tried it with our
ActionsDataTable specs. Tests run on my server 20% faster (43 seconds instead of 53 seconds) and screenshots look much nicer and is "closer" to a real world result. Eg on PhantomJS 1.9.X the sort arrow icons in dataTables are misplaced.
Used PhantomJS from here: https://launchpad.net/~tanguy-patte/+archive/ubuntu/phantomjs
Was installed quickly and worked out of the box so far. Not sure about other possible problems. It says it is
FYI: With PhantomJS 2 we still get the crashes / seg faults
Related issue on Travis Github repository: https://github.com/travis-ci/travis-ci/issues/3225
Adding PhantomJS 2.0 could be done like this: https://github.com/piwik/piwik/compare/phantomjs_2.0?expand=1
2.0.1-developmentwhile the one for Travis is
2.0.0. As the version number is close to
2.0.0there could be differences in the captured screenshots between
2.0-1-developmentmeaning we might not be able to generate the same screenshots in local dev environment vs travis. Not sure if this is a problem
FYI: PhantomJS 2.0 executed in 31 minutes instead of 37 minutes even though all screenshots fail (it takes more time when screenshots fail usually). It's hard to compare it especially since it also depends on the Travis VM.
Problem: It failed at the end when generating the diff file like this: https://travis-ci.org/piwik/piwik/jobs/67782618#L6092 and https://travis-ci.org/piwik/piwik/jobs/67781956#L5203
Here's more output https://travis-ci.org/piwik/piwik/jobs/67929639#L7500 It fails when copying the expected screenshot here: https://github.com/piwik/piwik/blob/2.14.0-b7/tests/lib/screenshot-testing/support/diff-viewer.js#L56 from
/home/travis/build/piwik/piwik/tests/UI/screenshot-diffs/UIIntegrationTest_referrers_campaigns.png. As all UI tests fail we might just spawn too many child processes?
Not sure how to continue with this issue. Is it worth to investigate this buffer overflow? Do we want to use PhantomJS 2 or not? (see previously mentioned problems/advantages)
for example: http://builds-artifacts.piwik.org/ui-tests.phantomjs_2.0/13623.7/EvolutionGraph_annotation_edit_submit http://builds-artifacts.piwik.org/ui-tests.phantomjs_2.0/13623.7/EvolutionGraph_initial http://builds-artifacts.piwik.org/ui-tests.phantomjs_2.0/13623.7/Login_login_fail http://builds-artifacts.piwik.org/ui-tests.phantomjs_2.0/13623.7/Menus_admin_changed http://builds-artifacts.piwik.org/ui-tests.phantomjs_2.0/13623.7/ActionsDataTable_column_sorted
@diosmosis What do you reckon how much work it would be to remove PhantomJS backend and try using Electron? Not supporting different backends, just Electron? Did you have a look whether Electron provides all API's that we need? Eg inject CSS class, evaluate code, spawn processes (I presume we'd use node.js built-in here), ... Not sure re: https://github.com/atom/electron/issues/228
Background: I just spent 2 or 3 hours trying to make this UI test work until I noticed PhantomJS just doesn't handle the floating correctly and I haven't found a way yet to make it work http://builds-artifacts.piwik.org/ui-tests.7822_3/13859.7/UIIntegrationTest_goals_overview
Also with PhantomJS 2 I don't get this test working. This morning I was also 2 or 3 hours busy trying to fix another UI test and I couldn't.
I really think it's time to try something else if it isn't too much work. I think we use too many PhantomJS API's
I just installed electron and executed it and got the following message:
vagrant<a class='mention' href='https://github.com/piwik'>@piwik</a>-trusty64:~/www/piwik$ electron /usr/lib/node_modules/electron-prebuilt/dist/electron: error while loading shared libraries: libgtk-x11-2.0.so.0: cannot open shared object file: No such file or directory
The x11 requirement might be a problem since one doesn't really want to install x11 on a server or on travis (could be also slow to install maybe?)
I am currently running a serie of CasperJS tests with SlimerJS (Gecko, natively not headless) on a Jenkins box that takes screenshots of my application. This is possible thanks to
xvfb, and it works quite good (with some tweaks, as always...). So I guess running Electron should be also possible, I'm really looking into it right now.
I hope I helped!
Thx for that! Do you get SlimerJS to run on Travis? From what I remember it's not headless?
So we are running SlimerJS tests in a Jenkin build, but it just launches a task on a AWS box... Anyhow, on Travis you have several other options: http://docs.travis-ci.com/user/gui-and-headless-browsers/
All of these are pretty great, but I think Saucelab is the way to go for an OpenSource project, as it's free!
What do you reckon how much work it would be to remove PhantomJS backend and try using Electron? Not supporting different backends, just Electron?
I already managed to get it working, until right before the actual tests run. That is to say, fixtures can be run successfully, but I didn't convert page-renderer.js.
Did you have a look whether Electron provides all API's that we need? Eg inject CSS class, evaluate code, spawn processes (I presume we'd use node.js built-in here), ... Not sure re: atom/electron#228
$('...').click() instead of
sendMouseEvent()). Tests will lose a little accuracy though (eg, if an animation is broken so the click won't work in a real browser, we won't notice because we just trigger the handler directly).
Btw, I don't know why "inject CSS class" would be a feature, it's just a evaluate JS use case.
Regarding lack of headless-ness,
xvfb should work fine. We used it w/ phantomjs for a bit in travis while experimenting w/ slimerjs.
FYI: I quickly tried to use slimerjs but a problem is that global variables do not work there see https://docs.slimerjs.org/current/differences-with-phantomjs.html
I will invest a few minutes to try to make it run but probably it'll be a bit more work.
I can try to help you, I migrated 2 or 3 times in the past month PhantomJS tests to SlimerJS, and the last time was yesterday :p
I spent 40 minutes on it but everything in our screenshot testing lib is basically based on global variables and we use quite a lot of
phantom.injectJS to have variables available globally which doesn't work and it is expected (https://github.com/laurentj/slimerjs/issues/337). We'd have to rewrite to use modules.
require('fs').changeWorkingDirectory() did not work but that should be maybe possible via
process. It was not needed for my test. I reckon it's about 1 day of work to make it run with SlimerJS. @mattab what do you think is it worth to give it a try?
@ngotchac Help is much appreciated. I'll definitely ping you when there are problems.
@tsteur if there is a chance we can make SlimerJS work in one day, +1 to try it
After some more thoughts and discussion I think the long term best solution might be electron (from what it seems). It recently added support for webdriver which is pretty awesome see http://electron.atom.io/docs/v0.28.0/tutorial/using-selenium-and-webdriver/ and https://code.google.com/p/selenium/wiki/WebDriverJs
We can use all the feature of node.js, it is actually maintained and updated, it is stable and should be fast, bigger community, many tools around electron etc. The only downside is the need of
X11 so far. Think I changed my mind and would rather prefer to give that one a try
Just FYI: Today I spent again 4 hours trying to fix 5 segment selector tests (http://builds-artifacts.piwik.org/ui-tests.7822_3/13908.7/) that started to fail after this commit: https://github.com/piwik/piwik/commit/7b6132d253fa95b025f16e02b39d7669ff941f36
console.log in UI tests etc. What makes the test work seems to
click() on the element directly (https://github.com/piwik/piwik/commit/729af37bf4810f8d7ec2d7f2f26c07bcd0998cf2 ). This would mean there is maybe an invisible div on top of the
Save button but it is basically impossible to find it. I just presume it is a problem with PhantomJS
As soon as possible I'll have a look re an alternative
I tried to make it work with Electron today but looks like we definitely need WebDriver, couldn't make it work without. Eg for clicking, reading content of page, manipulating page, ...
It would be maybe something good to work on in Berlin or Poland with combined power, will see :)
What makes electron very interesting is https://github.com/segmentio/nightmare which is like CasperJS but for electron.
We likely update to PhantomJS 2. We're also testing https://github.com/webdriverio/webdrivercss + https://github.com/webdriverio/webdriverio but we need to wait for webdriverio 4 support in webdrivercss (refs https://github.com/webdriverio/webdrivercss/issues/123 https://github.com/webdriverio/webdrivercss/issues/151 )
BTW: In future Puppeteer might be very interesting see https://github.com/GoogleChrome/puppeteer and https://ropig.com/blog/end-end-tests-dont-suck-puppeteer/
It looks like this could work quite nicely and easy to build / migrate etc.