Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch UI tests from phantomjs to chrome headless #14421

Merged
merged 137 commits into from May 8, 2019
Merged

Conversation

tsteur
Copy link
Member

@tsteur tsteur commented May 6, 2019

Installing chrome headless:

$ cd /path/to/matomo/tests/lib/screenshot-testing
$ npm install

Then execute tests:run-ui as usual.

Chromium Quirks

Generally to write new tests or convert existing tests, you can look at some of the already converted tests and copy the format. It's pretty straightforward and shouldn't take too long to figure out.

Note: the comparison threshold feature is still there, but it no longer uses resemblejs since that library can't be used synchronously (well, not easily). There's an example of the use of comparison threshold in UsersManager_spec.js.

Chromium does, however, have some quirks it helps to be aware of, including:

  1. if inputs are focused when taking screenshots, tests will fail randomly since the cursor can appear in the screenshot
  2. taking screenshots of modals is especially annoying as they will fail randomly in many cases. to reduce the randomness I've found it works to add wait times to make sure animations/css transitions complete (though we could potentially turn all transitions off during UI tests w/ !important), to wait for elements in the modal to become visible, and to experiment between const e = await page.$('.modal'); await e.screenshot(); vs await page.screenshotSelector('.modal'); (sometimes one will work, sometimes the other; usually the first format should be favored since it depends on puppeteer's code and not our own).
  3. since chrome is pretty fast, you'll have to sometimes do multiple waits, especially on page loads/reloads. For example, when loading the main page w/ dashboard, we await page.waitForNetworkIdle(); to make sure as many http requests are completed at first (so we don't take a screenshot before the page load occurs if there's a widget on the screen), then we await page.waitFor('.widget', { visible: true }); to make sure a widget is visible (otherwise the screenshot could be taken before the widget HTTP requests are sent), then wait we await page.waitForNetworkIdle(); because not all of the widgets may have completed loading. It takes a bit of experimentation for each test, but once figured out the tests seem to pass predictably.

diosmosis and others added 30 commits April 22, 2019 16:24
@diosmosis diosmosis changed the title Chrome headless3 Switch UI tests from phantomjs to chrome headless May 8, 2019
@diosmosis diosmosis added this to the 3.10.0 milestone May 8, 2019
@diosmosis diosmosis added the c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. label May 8, 2019
@diosmosis diosmosis merged commit 76e6059 into 3.x-dev May 8, 2019
@mattab mattab deleted the chrome-headless3 branch June 29, 2019 12:53
@mattab
Copy link
Member

mattab commented Jun 29, 2019

Nice work! @diosmosis @sgiehl 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants