As a developer I want to have a faster response whether my code breaks any tests #6429

mnapoli · 2014-10-13T01:34:43Z

Usually one would run the tests simply on the local server before committing it. Actual problem is our tests are very slow. For most developers it can take up to an hour for the tests to finish. Running the UI tests might take even longer. This is why we usually commit often to let Travis run the tests and to see whether we break anything.

As said we currently run our tests on Travis but it can take up to hours to get feedback there. Especially when there are many commits at the same time and resulting the jobs are queued. Even waiting 10 minutes for the tests to start on Travis is not acceptable as it means we have to switch to another task until I get a result (10 min waiting time + up to 20min execution time).

Following #6412 (Getting build feedback from CI is too slow) a possible solution would be to:

Use Travis CI to run the full test suite on PHP 5.3 only
Run the full test suite on all the other PHP versions + code coverage outside of Travis, in an Amazon EC2 instance for example
Currently, the Travis build takes 36 minutes (see https://ci-status.com) plus the queue time. In theory, that time would be divided by 4, so a little less than 10 minutes. That would also mean much less queue time.

This reduces the waiting time a bit but as there are many developers working on this projects we will often still have to wait for a test to start etc. Travis does NOT scale as we cannot have more concurrent jobs.

The idea to solve this is to have a console command that starts the tests immediately on an AWS instance. This means no waiting time and possibly the tests execute faster as on travis. It would be nice to have the following features:

Run tests against a specific git hash, branch, tag
Apply a patch on top of the checked out version so one does not have to commit to test anything
Screenshot diffs for UI tests

We have evaluated for instance JoliCI but it doesn't support the same features as Travis does. For instance there is no database and there are no notifications. Otherwise it is a cool tool!

Update:
We have a new command tests:run-aws that launches an instances based on a custom AMI. There the tests are executed immediately so no waiting time! As expected the execution time is much faster compared to Travis:
Integration Tests: 3:40 minutes on Travis, 1:40 on AWS
System Tests: 16 minutes on Travis, under 8 minutes on AWS
UI Tests: 21 minutes on Travis, 15 minutes on AWS

On top we save the waiting time for Travis to configure the virtual machine which saves as 1-3 minutes.

Later we could have different AMIs for different PHP versions etc.

To further improve test response we could at some point to start multiple phpunits and execute tests in parallel. Maybe a project that does this already exists. Anyway, we would have to use multiple databases etc which is not supported by Piwik yet.

The text was updated successfully, but these errors were encountered:

diosmosis · 2014-10-13T21:11:43Z

Any thoughts about using drone: https://github.com/drone/drone & http://drone.io ?

I don't know what it will take to get it to work w/ .travis.yml files.

mattab · 2014-10-13T22:45:57Z

drone.io does not look cheap as getting 8 concurrent builds already costs 400 USD...

diosmosis · 2014-10-13T22:51:46Z

Its open source I think, see the GitHub link.

…te file

…e same time as on travis although each test seems to be 30% aster. In total it seems to be still the same time

…will get a screenshot diff. Took 15min to execute UI tests compared to about 19 min on Travis

…. Fixed naming of command

mnapoli · 2014-10-28T22:25:13Z

Maybe we should close this issue? JoliCI can't run our travis.yml anyway because it's too complex.

…ays have to commit changes in order to execute them

tsteur · 2014-10-28T22:43:06Z

I'm still working on it. Will update issue later

…space

…e we run a test as the UI test overwrites the stored config.ini. Otherwise it is not possible to run first a UI test and then a system test. On the other side it allows us easily to change config entries and run tests again

…ql data dir to RAM now to speed up system tests from 16min to under 8 minutes and integration tests from 3 minutes to under 1:30min. I wanted to do this in the AMI but it seems not to persist it so we have to do it when the instances launches.

tsteur · 2014-10-29T23:52:29Z

Please note the following UI tests always fail on AWS as the screenshots contain for instance a path which is different to the one on travis:

Installation_system_check
UIIntegrationTest_admin_manage_tracking_code
UIIntegrationTest_admin_plugins Expected    
UIIntegrationTest_admin_security_info   
UIIntegrationTest_admin_visitor_generator
UIIntegrationTest_db_connect_error
Updater_main

Either we update the puppet at some point to have another user travis and same file structure (lot of work) or we fix the screenshot tests somehow to not print paths or we just leave it like this for a while

diosmosis · 2014-10-30T01:52:29Z

We can manually remove the paths from screenshot tests by grepping through the HTML and looking for PIWIK_INCLUDE_PATH and replacing w/ '/'. Should be enough I think.

tsteur · 2014-10-30T03:37:49Z

Good idea. I'll create an issue for this.

tsteur · 2014-10-30T04:26:00Z

I am going to close this for now. All future changes can be handled in new issues.

mattab · 2014-10-30T04:45:41Z

Great work guys!

…s in case it is a system test

mnapoli added Task Indicates an issue is neither a feature nor a bug and it's purely a "technical" change. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. labels Oct 13, 2014

mattab mentioned this issue Oct 13, 2014

Getting build feedback from CI is too slow -> let's make it faster #6412

Closed

4 tasks

tsteur added a commit that referenced this issue Oct 28, 2014

refs #6429 added a command to trigger tests on aws

a6b2a6d

tsteur added a commit that referenced this issue Oct 28, 2014

refs #6429 the namespace correction got lost when squashing the commits

b251922

tsteur mentioned this issue Oct 28, 2014

Added a test runner that launches tests on AWS #6540

Merged

tsteur added a commit that referenced this issue Oct 28, 2014

refs #6429 enable Test Runner by default for now

78d98d5

tsteur added a commit that referenced this issue Oct 28, 2014

refs #6429 as 2.9.0-b1 is not released yet we can reuse existing upda…

f4f084a

…te file

tsteur added a commit that referenced this issue Oct 28, 2014

refs #6429 this makes UI tests already much faster. It still takes th…

41ae0af

…e same time as on travis although each test seems to be 30% aster. In total it seems to be still the same time

tsteur added a commit that referenced this issue Oct 28, 2014

refs #6429 assuming artifacts makes test execution 25% faster and we …

e52c0cd

…will get a screenshot diff. Took 15min to execute UI tests compared to about 19 min on Travis

tsteur added a commit that referenced this issue Oct 28, 2014

refs #6429 make sure to update git submodules after checkout git hash…

6f5e08f

…. Fixed naming of command

tsteur added a commit that referenced this issue Oct 28, 2014

refs #6429 added the possibility to apply a patch so one does not alw…

c038747

…ays have to commit changes in order to execute them

tsteur added a commit that referenced this issue Oct 28, 2014

refs #6429 update description before fixing the ui test

6bb634c

tsteur added a commit that referenced this issue Oct 28, 2014

refs #6429 moved code coverage generator to correct plugin

afce3ea

tsteur added a commit that referenced this issue Oct 28, 2014

refs #6429 my changes were not added... make sure to use correct name…

4b76c6b

…space

tsteur added a commit that referenced this issue Oct 29, 2014

refs #6429 fix a ui test as there is a new testrunner plugin

ac33e5a

tsteur mentioned this issue Oct 29, 2014

Remove TestRunner plugin in latest.zip #6547

Closed

tsteur changed the title ~~Speed up Travis by running some tests on Amazon EC2~~ As a developer I want to have a faster response whether my code breaks any tests Oct 29, 2014

tsteur mentioned this issue Oct 30, 2014

Make UI tests work on all servers #6558

Open

tsteur added a commit that referenced this issue Oct 30, 2014

refs #6429 added some docs, especially how to create a new AMI

b849958

tsteur closed this as completed Oct 30, 2014

tsteur added a commit that referenced this issue Nov 3, 2014

refs #6429 generate a tar package containing all the processed report…

300a2d8

…s in case it is a system test

tsteur added a commit that referenced this issue Nov 3, 2014

refs #6429 clean submodules before fetching any data [skip-ci]

0de3ff0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

As a developer I want to have a faster response whether my code breaks any tests #6429

As a developer I want to have a faster response whether my code breaks any tests #6429

mnapoli commented Oct 13, 2014

diosmosis commented Oct 13, 2014

mattab commented Oct 13, 2014

diosmosis commented Oct 13, 2014

mnapoli commented Oct 28, 2014

tsteur commented Oct 28, 2014

tsteur commented Oct 29, 2014

diosmosis commented Oct 30, 2014

tsteur commented Oct 30, 2014

tsteur commented Oct 30, 2014

mattab commented Oct 30, 2014

As a developer I want to have a faster response whether my code breaks any tests #6429

As a developer I want to have a faster response whether my code breaks any tests #6429

Comments

mnapoli commented Oct 13, 2014

diosmosis commented Oct 13, 2014

mattab commented Oct 13, 2014

diosmosis commented Oct 13, 2014

mnapoli commented Oct 28, 2014

tsteur commented Oct 28, 2014

tsteur commented Oct 29, 2014

diosmosis commented Oct 30, 2014

tsteur commented Oct 30, 2014

tsteur commented Oct 30, 2014

mattab commented Oct 30, 2014