Use new travis infrastructure for travis tests #8835

diosmosis · 2015-09-22T11:04:52Z

I got the tests to run and pass: https://travis-ci.org/piwik/piwik/jobs/81561860

They aren't much faster, but I have noticed the builds start much quicker (as in, from travis status created -> booting -> running).

TODO:

Fix UI tests where URL is different due to port being used.
Create ability for plugins to install their own packages in generated .travis.yml.
Make sure sudo: false is specified through command line parameter to generate so plugin builds can be converted one by one in case they need extra time.
~~Test against mariadb instead of mysql 5.6.~~ Test against mysql 5.5 for now, switch to 5.5 + mariadb in another ticket.
Test that plugin builds can pass w/ & w/o --sudo-false option.
Document new options & YAML partials.
Review & merge.

List of Changes

APT packages required by Piwik core in travis are added via travis' addons section. Plugins can extend this section by creating addons.apt.sources and/or addons.apt.packages files in their tests/travis directory. Doing so is only required if using the new travis infrastructure, since sudo will not be available.
Added a new switch to generate:travis-yml: --sudo-false. This will generate .travis.yml w/ sudo: false, which will use travis' new infrastructure. We don't do this by default because some plugin builds will require sudo (ie, LoginLdap).
Added a new DI environment, ui-test. ui-test.php DI config files will be included (along w/ test.php) during UI tests. Currently this environment is used to make sure the UI tests will pass no matter what port Piwik is hosted on (required since sudo rights are needed to listen to ports 1-1024 on linux).
Added [tests] port to let Piwik know it is on a port during tests.
Some bug fixes in the order in which DI configs are included in ContainerFactory.php. (User config was added before environment, should be after so it can override anything. Also, made it possible for plugins to specify dev.php DI configs.)
UI tests now automatically remove PIWIK_INCLUDE_PATH from the HTML so more tests will pass locally. The root URL is also removed from UI test HTML & from system test responses (which lets ScheduledReports tests pass when the port is not 80).
Make sure a symbolic link is created for the misc folder during UI tests. It's used by some tests.
In travis-scripts, fixed a bug that affected plugins that used custom build steps. The YAML partials were not indented correctly, now they are.
Travis builds will no longer test against MySQL 5.6. It is currently not possible to install 5.6 using travis' new infrastructure. Since we will be including tests against MariaDB soon-ish, shouldn't be too much of an issue.

Merge Strategy

Should be merged on a weekend so no one will need the build immediately.

Merge Changes to travis generation command + travis scripts to use new travis infrastructure travis-scripts#14
Merge this PR
Get build to pass and ensure plugin builds pass.

mattab · 2015-09-22T14:03:46Z

Nice progress, I think it will save quite a few build time minutes per day!

…ameter for port (since using port 80 is not allowed for normal users on linux).

…x issue.

… be generated on travis.

…n tests:run-ui.

…l command.

…ication.

diosmosis · 2015-10-27T08:47:32Z

This PR is ready for review. I don't think it will be a very fun review, though.

tsteur · 2015-10-27T20:38:22Z

config/environment/ui-test.php

+            // Overlay needs the full URLs in order to find the links in the embedded page (otherwise the %
+            // tooltips don't show up)
+            if (!empty($request['method'])
+                && $request['method'] == 'Overlay.getFollowingPages'


Should this be maybe done in the overlay plugin config ui-test.php? (not sure if it is possible and whether we look for ui-test.php environment in plugins)

I can add a ui-test.php file to Overlay. I can't add the event there though, since we'd need to abort the event in config/ui-test.php with the event in Overlay. I could create a test class UITestResponseManipulator that is used in this global event and make it use a blacklist. Then in Overlay, the ui-test.php could just add to the blacklist. Do you think this is a good idea?

Sorry, I don't really understand re aborting the event etc. If it's too much work should be fine to leave it here I reckon

I thought up a simpler way of doing this, will give it a shot

…normalizing, use DI to store whitelist and override in ui-test.php DI config for Overlay.

…hard-coded root URL.

diosmosis · 2015-10-29T13:58:00Z

Ready for another review. If there are no problems, will merge on weekend.

…tests.

tsteur · 2015-10-29T20:12:14Z

👍 lgtm

…n the list, not running code only for those entries.

Refs matomo-org/matomo#8835, changes to travis generation command + travis scripts to use new travis infrastructure.

Use new travis infrastructure for travis tests

diosmosis added Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. labels Sep 22, 2015

diosmosis self-assigned this Sep 22, 2015

diosmosis added this to the 2.15.0 milestone Sep 22, 2015

mattab modified the milestones: 2.15.1, 2.15.0 Oct 2, 2015

diosmosis force-pushed the travis_sudo_false2 branch 3 times, most recently from 0e2b8b1 to 510f063 Compare October 17, 2015 02:27

diosmosis added 20 commits October 26, 2015 21:45

Disable travis autoupdate for now.

7be17db

Enable container based travis build + remove python 2.6 install script.

34b402d

Try to install mysql 5.6 on container infrastructure.

c485da3

Disable some travis jobs for now.

32465e5

Use full nginx config instead of trying to modify system nginx config.

cfd8250

Try to fix nginx setup on travis. Includes new [tests] INI config par…

6521e98

…ameter for port (since using port 80 is not allowed for normal users on linux).

Switch back to sudo required momentarily to try and debug travis ngin…

df1d128

…x issue.

Try to fix nginx config.

55e13fb

Debugging travis test failure.

2a03a34

Add port in tests to tracker URL.

ac4245f

Fixing integration tests.

4b79d41

Remove Piwik URL from system test results before comparing.

0a2ca3d

Re-enable full build.

ef92031

Remove last instance of sudo in travis scripts.

824a6e6

Try to fix javascript + angular tests on travis.

be599ea

Debugging UI test failure.

0732036

More travis debugging.

779fad4

Make travis.sh go through tests:run-ui for UI tests so config.js will…

8f08bcc

… be generated on travis.

Add missing params to tests:run-ui.

3e252d5

Try to copy test fonts w/o sudo rights. And use correct return code i…

af70018

…n tests:run-ui.

diosmosis added 2 commits October 26, 2015 21:48

Update tests/travis submodule for new --sudo-false generate:travis-ym…

698ab83

…l command.

Update submodule after rebase.

ea304e5

diosmosis force-pushed the travis_sudo_false2 branch from 5f5f4f3 to ea304e5 Compare October 27, 2015 04:49

diosmosis added 2 commits October 26, 2015 21:51

Regenerate core .travis.yml & see if autoupdate works.

36e619f

Regenerate again.

6ba1314

diosmosis force-pushed the travis_sudo_false2 branch from 00ce047 to 6ba1314 Compare October 27, 2015 05:02

diosmosis mentioned this pull request Oct 27, 2015

Changes to travis generation command + travis scripts to use new travis infrastructure matomo-org/travis-scripts#14

Merged

Updating tests/travis submodule.

07a1609

diosmosis force-pushed the travis_sudo_false2 branch from 539da89 to 07a1609 Compare October 27, 2015 07:00

Remove trusted host change and try to fix w/ ui-test.php config modif…

cd9f9da

…ication.

diosmosis added Needs Review PRs that need a code review and removed Pull Request WIP Indicates the current pull request is still work in progress and not ready yet for a review. labels Oct 27, 2015

tsteur reviewed Oct 27, 2015
View reviewed changes

diosmosis added 3 commits October 29, 2015 04:48

Instead of hardcoding whitelisting of Overlay methods in UI test URL …

f1dcc9b

…normalizing, use DI to store whitelist and override in ui-test.php DI config for Overlay.

Do not remove URLs in API output during tests, instead replace it w/ …

ddf3911

…hard-coded root URL.

Use configured REMOTE_ADDR value in DB connection failure UI test.

9012148

Forgot to add example .yml files to ExamplePlugin for travis-scripts …

0199c3b

…tests.

Rename whitelist to blacklist since we're ignoring entries that are i…

798fcf1

…n the list, not running code only for those entries.

diosmosis added a commit to matomo-org/travis-scripts that referenced this pull request Oct 30, 2015

Merge pull request #14 from piwik/travis_sudo_false2

44793a8

Refs matomo-org/matomo#8835, changes to travis generation command + travis scripts to use new travis infrastructure.

Update tests/travis submodule.

64536b6

diosmosis added a commit that referenced this pull request Oct 30, 2015

Merge pull request #8835 from piwik/travis_sudo_false2

c975e0d

Use new travis infrastructure for travis tests

diosmosis merged commit c975e0d into master Oct 30, 2015

diosmosis deleted the travis_sudo_false2 branch October 30, 2015 20:12

mattab mentioned this pull request Nov 2, 2015

Use sudo:true Travis config #9146

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use new travis infrastructure for travis tests #8835

Use new travis infrastructure for travis tests #8835

diosmosis commented Sep 22, 2015

mattab commented Sep 22, 2015

diosmosis commented Oct 27, 2015

tsteur Oct 27, 2015

diosmosis Oct 28, 2015

tsteur Oct 28, 2015

diosmosis Oct 29, 2015

diosmosis commented Oct 29, 2015

tsteur commented Oct 29, 2015

Use new travis infrastructure for travis tests #8835

Use new travis infrastructure for travis tests #8835

Conversation

diosmosis commented Sep 22, 2015

List of Changes

Merge Strategy

mattab commented Sep 22, 2015

diosmosis commented Oct 27, 2015

tsteur Oct 27, 2015

Choose a reason for hiding this comment

diosmosis Oct 28, 2015

Choose a reason for hiding this comment

tsteur Oct 28, 2015

Choose a reason for hiding this comment

diosmosis Oct 29, 2015

Choose a reason for hiding this comment

diosmosis commented Oct 29, 2015

tsteur commented Oct 29, 2015