Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Converting UI tests from phantomjs to headless chrome #13048

Closed
wants to merge 69 commits into from

Conversation

diosmosis
Copy link
Member

Work in progress, more to follow.

@diosmosis diosmosis added Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. labels Jun 10, 2018
@fdellwing
Copy link
Contributor

Will this PR make #13036 work in tests?

@diosmosis
Copy link
Member Author

Commented on the other issue. This PR probably won't be finished for a while unfortunately :)

@sgiehl
Copy link
Member

sgiehl commented Jun 24, 2018

@diosmosis do you need help finishing this PR? Could maybe help converting some tests when I have some time in between...

@diosmosis
Copy link
Member Author

diosmosis commented Jun 24, 2018

I was actually doing this in my spare time but it took longer than expected (was hoping to finish in a weekend), and now I don't have the spare time. Would be happy for the help, feel free to push directly to the branch if you'd like @sgiehl!

Note: in order to run the tests, you should just have to do an npm install (w/ node 8 or above) in the tests/lib/screenshot-testing dir, then just run ./console tests:run ... as usual. Still need to have the right fonts & everything. On a mac I can only run them locally through docker (well, run them & have the tests pass).

expect.screenshot('column_sorted').to.be.capture(function (page) {
page.click('th#avg_time_on_page', 3000);
}, done);
await page.click('th#avg_time_on_page', 3000);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, the wait timeout no longer exists w/ chrome headless, instead you have to call one of the wait functions: https://github.com/GoogleChrome/puppeteer/blob/v1.4.0/docs/api.md#pagewaitforselectororfunctionortimeout-options-args

I added one extra one waitForNetworkIdle() that waits for network requests to finish. So here it would be:

await page.click('th#avg_time_on_page');
await page.waitForNetworkIdle();

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also you can wait for times via waitFor(1000), but have been trying to replace these w/ waitFor('.selector') unless its to wait out an animation. (Though in many cases they're unnecessary to have w/ chrome, and in other cases they can be reduced greatly since chrome is so much faster than phantomjs.)

@sgiehl
Copy link
Member

sgiehl commented Jul 17, 2018

@diosmosis I've rebased the branch on latest 3.x-dev and added more conversions. Most core tests are now converted, but there are some tests failing due to failing requests and I couldn't find out why. I've added the last commit to fix some failing requests for the logos, but that one should be removed again as soon as we know why the requests were failing in the first place. Maybe you have an idea.

@sgiehl
Copy link
Member

sgiehl commented Jan 5, 2019

@diosmosis I've rebased and update this branch with latest changes on 3.x-dev. ~90% of tests are already passing, but there's still some work left. Maybe you have some time in between to work on some of the stuff:

  • Installation, Login & TwoFactorAuth tests are currently skipped in a11d7bb as puppeteer completely hangs (rans into travis 10min timeout if returns are removed)
  • I've disabled the dangerous link check in 2fd9369 as it does not yet work again
  • Diffs aren't uploaded to artifacts server, and thus the failures aren't shown there
  • Trackingfailures tests don't seem to be able to generate the failures anymore
  • TagManager tests are not yet done
  • Some tests are still randomly failing as they slightly differ from time to time. Currently only a difference of 1 pixel is allowed, maybe we need to reimplement the comparison threshold.

@diosmosis
Copy link
Member Author

👍 will make some time for it. It's very annoying to have UI tests that never consistently pass, hope this can be merged soon.

@diosmosis diosmosis self-assigned this Jan 5, 2019
@diosmosis
Copy link
Member Author

The installation test failure is due to puppeteer/puppeteer#3471. W/o request interception it works. There was a recent commit in puppeteer to try and resolve this, but it doesn't seem to help in this case.

@sgiehl
Copy link
Member

sgiehl commented May 9, 2019

was done with #14421

@sgiehl sgiehl closed this May 9, 2019
@sgiehl sgiehl deleted the chrome-headless branch May 9, 2019 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants