When executing UI tests on Travis we do not execute the UI tests that are defined in the submodule plugins. On the contrary, we do execute unit, integration and system tests of these plugins. I think we should be consistent and also include UI tests of plugins that are added in
piwik/piwik as a submodule.
When working on these plugins it makes it easier to see if the UI tests of this plugin actually work and when working on Piwik we can see if we break any of these plugins. The ones that are added are some kind of special plugins that are kinda core but we want to provide updates separately etc.
Of course we also have Travis jobs in the plugins that are submodules. The problem is that they run against beta versions of Piwik. This means to make tests work for a plugin we need to release a new beta after each change in Piwik, and then change the
PIWIK_LATEST_STABLE_TEST_TARGET variable in
.travis.yml of the plugin.
I am currently developing the
CustomDimensions plugin and I cannot let Travis test the UI tests. To test the UI tests in
CustomDimensions Travis we would need to release a new beta version of Piwik. To release a new beta that can be used by this plugin we would need to merge a PR in
piwik/piwik that includes some changes in Piwik core that are needed to make this plugin work. However, we cannot really merge since we don't know whether the UI tests work etc.
This issue is to discuss whether we should include these plugins in UI tests or not. And if want to include them, we should probably remove the
--core flag in https://github.com/piwik/travis-scripts/blob/ab737e8de918bfc06c1fc5234071f60e3be51a60/travis.sh#L76
I think we should be consistent and also include UI tests of plugins that are added in piwik/piwik as a submodule.
This sounds good to me, but I'm wondering why we didn't do it in the first place, maybe there will be some downside?
This issue is to discuss whether we should include these plugins in UI tests or not.
@diosmosis what do you think, maybe you remember why we didn't run UI tests of submodule'd plugins in our
piwik/piwik UI tests CI job?
I do not remember why.
Just gave it a try to remove the
--core. Submodule tests would then be executed. Currently a lot tests would be failing. See https://builds-artifacts.matomo.org/matomo-org/matomo/test/38431/
But what could cause problems is, that it won't be possible to fix the tests for QueuedTracking or LoginLdap plugin, as they require additional software to be installed and set up. Not sure if we should do that for all core builds then? 🤔
I reckon some complications will be that it might be hard to have the UI tests working in the submodule, and the core. Eg if the plugins runs using latest Matomo or so, and the core runs using latest
4.x-dev etc. It'll be hard to impossible to have them both green unless we start saving screenshots per
branch or version but this obviously comes with quite some overhead. Not sure it's fully worth it for now.
Guess that would only work smoothly if we run the plugin builds against latest 4.x-dev instead of latest release. Everything else isn't worth the overhead I think.
4.x-dev might be sometimes hard when making changes in core and the plugin for example. But I think this does currently not work anyway because the tests run against the max supported piwik. Which be the latest beta.
So it be actually an improvement maybe running it against 4.x-dev if I see this right?