Upgrade travis to trusty environment #12087
Conversation
|
Hi @mneudert Excellent progress 👍 I checked all the UI tests and most can be safely overriden, except for one issue. Most UI tests failures are due to:
Blocking issue: missing GD freetypeThe missing GD/freetype issue (visible in some of the ui tests such as this System check test) is the only blocking issue because we are testing that our static graphs generate correctly in the following tests:
-> is it possible to install the GD/freetype in Travis so these screenshot tests will execute as expected? |
mattab
added
the
c: Tests & QA
|
@mneudert tried to trigger a few builds but it failed due to wrong submodule. Don't really understand why your last commit triggered build OK while mine didn't 😕 Looking forward to merging this hopefully 👍 |
|
Hmpf, yeah. While testing some submodules ( For the GD changes I am searching for a solution. It would be quite unexpected to not be able to add the missing parts. Otherwise I would expect more users to mention the graphs are broken/not visible because of the same setup problems. |
|
@mneudert maybe we could contact Travis CI and ask them for help? They usually reply quickly and are helpful so we shouldn't hesitate in case they maybe know a solution or a tip. |
|
If we switch the UITests to So we could upgrade the PHP version or ask travis to change the installed PHP version to include freetype support. Or perhaps both :) Added a php-version-flip-commit to see how a full test run holds up to my observations... |
|
Not sure anymore whether lowering any of the test requirements to |
|
@mneudert fyi you can try change the requirements of the tests, so they run on Travis on 5.5. The idea behind the message is to make sure they at least run there. The reason they are restricted to one PHP version and GD is because GD has some minor pixel differences depending on the php/gd versions and we need to always run them on the same one (but we can change the versions and updated the images) |
|
Also there are a few randomly failing tests: 4 failures here1 failure hereSolution to randomly failing testsThese issues are caused by a difference in the sorting of elements. I think caused by this PHP sort behavior documented here
This could maybe be fixed by adding another sort where when elements have the same 'count', we could sort them in alphabetical order to ensure the order is constant and does not depend on the |
@mneudert sounds good to upgrade the PHP version for UI tests 👍 |
|
I tried modifying the SystemTests to run with Already reached out to Travis to see if that is intentional or could be resolved: travis-ci/travis-ci#8510 From what I see in your builds all flapping tests are includede in the |
|
Yes they're all tests in the customDimensions plugin. |
|
As it seems the missing FreeType support on PHP 5.5 for the new Travis infrastructure is a given. Not by intention but perhaps also not changed quickly. For now I switched the UITests to the old infrastructure in order to have a working PHP 5.5 build in the matrix because I really don't like dropping the low end requirement. Using the custom matrix we only have to change 2 lines to change the infrastructure. |
|
FYI: There's also this failing test in the last build: There was 1 failure:
1) Piwik\Plugins\SegmentEditor\tests\Integration\SegmentEditorTest::test_UpdateSegment
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
Array (
- 'idsegment' => 2
+ 'idsegment' => '2'
'name' => 'NEW name'
'definition' => 'searches==0'
'login' => 'superUserLogin'
- 'enable_all_users' => 0
- 'enable_only_idsite' => 0
- 'auto_archive' => 0
+ 'enable_all_users' => '0'
+ 'enable_only_idsite' => '0'
+ 'auto_archive' => '0'
'ts_created' => '2017-10-05 18:50:'
- 'ts_last_edit' => '2017-10-05 18:51:'
- 'deleted' => 0
+ 'ts_last_edit' => '2017-10-05 18:50:'
+ 'deleted' => '0'
)
@mneudert looking forward to merging this one (which requires https://github.com/piwik/plugin-CustomDimensions/issues/71 ) |
|
Race conditions ❤️ As far as I remember that error did not occur in any of the previous runs. As the As the update is done after generating the expected result values I can only think of modifying the result to expect not a specific time but a range check for something like "ts_created + 2 seconds". Will try something like that 👍 |
|
After reading properly I saw we are talking about Thinking about "assertTimeWithinSeconds($expected, $result, $seconds)" and removing it from the other result assertions... |
nice catch. I'd say we can easily ignore it as I don't remember seeing this race condition many times... |
|
Hi @mneudert would you mind updating the submodule? the build may be green this time and we can merge this useful PR 👍 |
|
Tests look rather good to me now 👍 When merging one should take care to remove the first two commits beforehand. After merging the |
@mneudert I'm not sure about removing commits from PR. Would you mind merging the PR yourself maybe? I've reviewed it and happy with it 👍 |
|
Hmpf, hit the wrong button on a different page. So tests need a second run to show their results here... (previous: https://travis-ci.org/piwik/piwik/builds/287273686) |
|
Failed screenshot tests look like some usual noise.
|
|
I gonna merge this now 🎉 |
Pull Request to integrate/validate changes made in matomo-org/travis-scripts#36.
The submodule change to the fork should obviously be ignored. The "bump submodules" commit integrates changes from the travis repository after running
./console generate:travis-yml --coreand should be replaced before merging with a proper submodule update.