@tsteur opened this Pull Request on May 6th 2019 Member

Installing chrome headless:

$ cd /path/to/matomo/tests/lib/screenshot-testing
$ npm install

Then execute tests:run-ui as usual.

Chromium Quirks

Generally to write new tests or convert existing tests, you can look at some of the already converted tests and copy the format. It's pretty straightforward and shouldn't take too long to figure out.

Note: the comparison threshold feature is still there, but it no longer uses resemblejs since that library can't be used synchronously (well, not easily). There's an example of the use of comparison threshold in UsersManager_spec.js.

Chromium does, however, have some quirks it helps to be aware of, including:

1) if inputs are focused when taking screenshots, tests will fail randomly since the cursor can appear in the screenshot
2) taking screenshots of modals is especially annoying as they will fail randomly in many cases. to reduce the randomness I've found it works to add wait times to make sure animations/css transitions complete (though we could potentially turn all transitions off during UI tests w/ !important), to wait for elements in the modal to become visible, and to experiment between const e = await page.$('.modal'); await e.screenshot(); vs await page.screenshotSelector('.modal'); (sometimes one will work, sometimes the other; usually the first format should be favored since it depends on puppeteer's code and not our own).
3) since chrome is pretty fast, you'll have to sometimes do multiple waits, especially on page loads/reloads. For example, when loading the main page w/ dashboard, we await page.waitForNetworkIdle(); to make sure as many http requests are completed at first (so we don't take a screenshot before the page load occurs if there's a widget on the screen), then we await page.waitFor('.widget', { visible: true }); to make sure a widget is visible (otherwise the screenshot could be taken before the widget HTTP requests are sent), then wait we await page.waitForNetworkIdle(); because not all of the widgets may have completed loading. It takes a bit of experimentation for each test, but once figured out the tests seem to pass predictably.

@mattab commented on June 29th 2019 Member

Nice work! @diosmosis @sgiehl :rocket:

This Pull Request was closed on May 8th 2019
Powered by GitHub Issue Mirror