The last big missing functionnality not yet tested is our Installation process, as well as the update process.
Let's add screenshot tests for:
The result will be that we always know whether installation + update to latest Piwik work. This will result in more agility and ability to release new versions faster.
It is really critical for our QA process to re-enable the Webtests on Jenkins.
@Anthon would you have an estimate for this task? is it half day or 2 days work? :)
Much more than 2, I suspect -- a lot of web tests were disabled.
Moreover, the ticket involves rewriting all the web tests.
(oh another option is phpspec + mink)
It sounds really hard and might never happen.. :(
Maybe we could find a workaround to re-enable at least the install/update tests which were an incredibly useful part of our QA toolset... ?
It might be nice to use Behat+Mink. I've already done work here to parallelize Selenium testing.
Webtests are not that critically important, compared to getting Jenkins back to working. Decreasing priority...
Anthon, do you reckon I should create a ticket for Jenkins setup?
Many times our sql update code wasn't failure-friendly and would fail when re-executed as it left the DB in a half-updated state.
Having the following test would ensure that all future update scripts written are working in a failure scenario.
A new idea for a webtest:
Download 1.8.3 release Download latest Run Auto update - but make sure the auto update code fails at the last step Re-run auto update: the code should still run and the SQL not fail. For this, some queries will throw exceptions which must be specifically ignored, eg. "1060 Duplicate column name 'xx'" can be ignored for an ALTER that adds a column. The update re-running after initial failure should succeed.
Consolidating milestones FTW
In 50742af6d5784d78bdf36281880c45310875a698: Refs #2935, add installation UI test, allow no fixture to be specified for screenshot test and fix following installation process regressions:
In f05c080ec3dc724929b6836ca4cfbea1aab36f0a: Refs #2935, fixing Installation UI test.
@capedfuzz well done on adding those tests. Please close the ticket if it's fixed :)