Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random UI test failures (due to bug in phantomjs2) #10536

Closed
mattab opened this issue Sep 22, 2016 · 3 comments
Closed

Random UI test failures (due to bug in phantomjs2) #10536

mattab opened this issue Sep 22, 2016 · 3 comments
Labels
answered For when a question was asked and we referred to forum or answered it. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues.

Comments

@mattab
Copy link
Member

mattab commented Sep 22, 2016

Often the visitor log UI tests, and overlay UI tests, are randomly failing. as we need to have a green build for UI tests, for now we must disable these tests. It's bad practise to disable UI tests but we have no choice if we want to see a green build for other practical reasons.

Ideally in the future phantomjs2 will find the solution to this random failure and we can re-enable these UI tests.

@mattab mattab added the c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. label Sep 22, 2016
@mattab mattab added this to the 3.0.0 milestone Sep 22, 2016
mattab added a commit that referenced this issue Sep 22, 2016
* Comment out Visitor Log UI tests refs #10536

* UI tests: config file was changed and fields were updated
@mattab
Copy link
Member Author

mattab commented Sep 22, 2016

This bug: ariya/phantomjs#12750

screenshot tests that often fail randomly are:

  • ActionsDataTable_pageview_percentages.png
  • ActionsDataTable_unique_pageview_percentages.png
  • UIIntegrationTest_dashboard3.png
  • UIIntegrationTest_admin_settings_general.png
  • Overlay_page_link_clicks.png
  • Overlay_page_new_links.png
  • Overlay_page_change.png
  • Overlay_period_change.png
  • Overlay_row_evolution.png
  • Overlay_transitions.png
  • ...

@mattab
Copy link
Member Author

mattab commented Sep 26, 2016

Note to self: would it be maybe possible to "Retry on failure" for those random UI test failures? Maybe there's a feature in moccha to retry failed tests...

mattab added a commit that referenced this issue Sep 29, 2016
* Fix depraction test: use assertDeprecatedMethodIsRemovedInPiwik3

* Fix Scheduled Reports sent one hour late in daylight saving timezones (#10443)

* convert hour to send report to/from UTC, to ensure it isn't affected by daylight savings

* adds update script to change existing scheduled reports to use utc time

* code improvement

* adds missing param

* Added new event Archiving.makeNewArchiverObject to  allow customising plugin archiving  (#10366)

* added hook to alllow plugin archiving prevention

* cr code style notes

* reworked PR to fit CR suggestions

* added PHPDoc for hook

* Event description more consistent

* UI tests: minor changes

* Comment out Visitor Log UI tests refs #10536

* Adds test checking if all screenshots are stored in lfs

* removed screenshots not stored in lfs

* readds screenshots to lfs

* 2.16.3-b4

* Issue translation updates against 2.x-dev

* language update

* Fix bug in widget list remove where the JSON object becomes array

* 2.16.3-rc1

* console command custom-piwik-js:update should work when directory is writable and file does not exist yet (#10576)

* followup #10449

* Fix test
(cherry picked from commit fac3d63)

* Prevent chmod(): No such file or directory

* Automatically update all marketplace plugins when updating Piwik (#10527)

* update plugins and piwik at the same time

* make sure plugins are updated with piwik

* use only one try/catch

* reload plugin information once it has been installed

* make sure to clear caches after an update

* fix ui tests

* make sure to use correct php version without any extras

* Additional informations passed in the hook "isExcludedVisit" (issue #10415) (#10564)

* Additional informations passed in the hook "isExcludedVisit" (issue #10415)

* Added better description to the new parameters

* Update VisitExcluded.php

* Remove two parameters not needed as better to use the Request object

* Update VisitExcluded.php

* remove extra two parameters in VisitExcluded constructor to prevent confusion (#10593)

* Update our libs to latest #10526

* Update composer libraries to latest #10526

* Update log analytics to latest

* When updating the config file failed (or when any other file is not writable...), the Updater (for core or plugins) will now automatically throw an error and cancel the update (#10423)

* When updating the config file failed (or when any other file is not writable...), the Updater (for core or plugins) will now automatically throw an error and cancel the update

* add integration test to check the correct exception is thrown when config not writabel

* New integration test for updater

* Make test better

* When opening the visitor profile, do not apply the segment (#10533)

* When opening the visitor profile, do not apply the segment

* added ui test for profile but does work for me

* next try to make ui test work

* add expected screenshot

* added missing doc
@mattab
Copy link
Member Author

mattab commented Feb 28, 2017

We've implemented retry on failure using Mocha retry, and it works well it seems

@mattab mattab closed this as completed Feb 28, 2017
@mattab mattab added the answered For when a question was asked and we referred to forum or answered it. label Feb 28, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
answered For when a question was asked and we referred to forum or answered it. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues.
Projects
None yet
Development

No branches or pull requests

1 participant