Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UI tests on Travis do not execute UI tests of submodules #9253

Closed
tsteur opened this issue Nov 23, 2015 · 7 comments · Fixed by #15844
Closed

UI tests on Travis do not execute UI tests of submodules #9253

tsteur opened this issue Nov 23, 2015 · 7 comments · Fixed by #15844
Assignees
@sgiehl
Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues.
Milestone
4.0.0

Comments

@tsteur
Copy link
Member

tsteur commented Nov 23, 2015

When executing UI tests on Travis we do not execute the UI tests that are defined in the submodule plugins. On the contrary, we do execute unit, integration and system tests of these plugins. I think we should be consistent and also include UI tests of plugins that are added in piwik/piwik as a submodule.

When working on these plugins it makes it easier to see if the UI tests of this plugin actually work and when working on Piwik we can see if we break any of these plugins. The ones that are added are some kind of special plugins that are kinda core but we want to provide updates separately etc.

Of course we also have Travis jobs in the plugins that are submodules. The problem is that they run against beta versions of Piwik. This means to make tests work for a plugin we need to release a new beta after each change in Piwik, and then change the PIWIK_LATEST_STABLE_TEST_TARGET variable in .travis.yml of the plugin.

I am currently developing the CustomDimensions plugin and I cannot let Travis test the UI tests. To test the UI tests in CustomDimensions Travis we would need to release a new beta version of Piwik. To release a new beta that can be used by this plugin we would need to merge a PR in piwik/piwik that includes some changes in Piwik core that are needed to make this plugin work. However, we cannot really merge since we don't know whether the UI tests work etc.

This issue is to discuss whether we should include these plugins in UI tests or not. And if want to include them, we should probably remove the --core flag in https://github.com/piwik/travis-scripts/blob/ab737e8de918bfc06c1fc5234071f60e3be51a60/travis.sh#L76
@mattab
Copy link
Member

mattab commented Nov 23, 2015

I think we should be consistent and also include UI tests of plugins that are added in piwik/piwik as a submodule.

This sounds good to me, but I'm wondering why we didn't do it in the first place, maybe there will be some downside?

This issue is to discuss whether we should include these plugins in UI tests or not.

@diosmosis what do you think, maybe you remember why we didn't run UI tests of submodule'd plugins in our piwik/piwik UI tests CI job?

@mattab mattab added the c: Tests & QA For issues related to automated tests or making it easier to QA & test issues. label Nov 23, 2015
@diosmosis
Copy link
Member

I do not remember why.

@mattab mattab added this to the 2.15.1 milestone Nov 25, 2015
@diosmosis diosmosis self-assigned this Nov 26, 2015
@mattab mattab modified the milestone: 2.15.1 Dec 4, 2015
@mattab mattab added this to the 3.0.0 milestone Dec 23, 2015
@sgiehl
Copy link
Member

sgiehl commented Feb 14, 2020

Just gave it a try to remove the --core. Submodule tests would then be executed. Currently a lot tests would be failing. See https://builds-artifacts.matomo.org/matomo-org/matomo/test/38431/

But what could cause problems is, that it won't be possible to fix the tests for QueuedTracking or LoginLdap plugin, as they require additional software to be installed and set up. Not sure if we should do that for all core builds then? 🤔

@tsteur
Copy link
Member Author

tsteur commented Feb 16, 2020

I reckon some complications will be that it might be hard to have the UI tests working in the submodule, and the core. Eg if the plugins runs using latest Matomo or so, and the core runs using latest 4.x-dev etc. It'll be hard to impossible to have them both green unless we start saving screenshots per branch or version but this obviously comes with quite some overhead. Not sure it's fully worth it for now.

@sgiehl
Copy link
Member

sgiehl commented Feb 17, 2020

Guess that would only work smoothly if we run the plugin builds against latest 4.x-dev instead of latest release. Everything else isn't worth the overhead I think.

@tsteur
Copy link
Member Author

tsteur commented Feb 17, 2020

4.x-dev might be sometimes hard when making changes in core and the plugin for example. But I think this does currently not work anyway because the tests run against the max supported piwik. Which be the latest beta.

So it be actually an improvement maybe running it against 4.x-dev if I see this right?

This was referenced Feb 23, 2020
Merged
Merged
Merged
Merged
@sgiehl
Copy link
Member

sgiehl commented Feb 25, 2020

All submodule plugin are now running their tests against 4.x-dev and minimum_required_piwik

@sgiehl sgiehl self-assigned this Apr 22, 2020
@sgiehl sgiehl mentioned this issue Apr 22, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sgiehl sgiehl

Labels
c: Tests & QA For issues related to automated tests or making it easier to QA & test issues.
Projects
None yet
Milestone
4.0.0
Development

Successfully merging a pull request may close this issue.

Run UI tests of submodules with core build if possible matomo-org/matomo
4 participants
@diosmosis @tsteur @mattab @sgiehl