Re-enable Dashboard UI tests #8163
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tsteur
added
c: Tests & QA
It looks like resemble causes the random crashes during UI tests on Travis CI. I can reproduce this locally and identified resemble as the problem. The tests crashed because of too much memory usage We now also avoid loading the images into PhantomJS to compare them. By executing ImageMagick we seem to save a lot of memory and time. Eg currently on master the tests need more than 50 minutes whereas with this solution it takes only approx 37 minutes.
tsteur
force-pushed
the
avoid_resemble_comparison_if_possible
branch
from
ce7d353
to
3d100bd
Compare
|
This PR also fixes an issue that we did not detect all UI regressions. Eg there was always a default tolerance in |
tsteur
added a commit
that referenced
this pull request
Jun 22, 2015
…ible Re-enable Dashboard UI tests
mattab
added
the
not-in-changelog
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #8142 #8034
It looks like resemble causes the random crashes during UI tests on Travis CI. I can reproduce this locally and identified resemble as the problem. The tests crashed because of too much memory usage.
We now avoid loading the images into PhantomJS to compare them and we use
resemble.jsonly as a fallback if there's a threshold defined (which is currently only the case for ImageGraph tests). The dashboard tests failed since the images are pretty long. By executing ImageMagick we seem to save a lot of memory and time. Eg currently on master the tests need more than 50 minutes whereas with this solution it takes only approx 37 minutes. This explains why the UI test build take longer the more failing tests there are as more pictures need to be compared using resemble (which is slow). So forgreentests the time difference might be actually much less.It should also fix a problem with resemble that we probably often not compared the whole image in tests. From the Resemble.js docs:
To fix this I do now set a
largeImageThresholdparameter.To improve performance of ImageMagick further I wanted to set an option to stop comparing as soon as there is at least one pixel difference but did not find such option.
FYI: https://developer.piwik.org/guides/tests-ui already mentions the requirement of ImageMagick. I was thinking about adding a check whether
comparecommand exists in./console tests:run-uibut decided not to as it I'm not sure re Windows etc. When compare does not exist there should be an error message once a screenshot is actually compared.I'm not sure if we should merge directly. Ideally we would make
mastergreen before merging this one but in most cases the tests onmastertake more than 50 minutes which makes this a bit hard.