one feedback is that fixing the tests every week is quite time consuming - maybe we could schedule to fix the builds of all plugins only once a month, for example ~ two weeks before the release

Goals

• Detect when a plugin is broken / its behavior is changed by a change in the stable Piwik platform
• Detect when a plugin is broken when running with the minimum required piwik version (as set in its plugin.json file)
• fixing the tests every week could be time consuming (we'd like to find a safe way to have to fix builds less often while meeting our high quality requirements)

Assumptions

• we ensure our plugins don't call non-public APIs, (requires: Remove and/or fix all uses of non-API & deprecated methods/fields/classes/interfaces in maintained plugins #7569 and Decide how we keep official plugins from calling non-public APIs or deprecated code #7633)
• we ensure our plugins don't call deprecated code. (requires: Remove and/or fix all uses of non-API & deprecated methods/fields/classes/interfaces in maintained plugins #7569 and Decide how we keep official plugins from calling non-public APIs or deprecated code #7633)

Summary: our plugins should never break or get broken by a newer Piwik version.

Proposal Plugin CI jobs setup

• (1) Plugin CI job running against minimum required Piwik would fail build when the CI job fails
  • this is current behavior, it allows us to detect when a plugin is broken when running with the minimum required piwik
• (2) Create a new plugin CI job running against latest stable Piwik that would fail the build if the job fails
• it would allow us to detect when a plugin is broken / its behavior is changed by a change in the stable Piwik platform
• (3) mark the Plugin CI jobs against Piwik platform master branch as Allowed failure.
• eg. if bugs are introduced in Piwik master, or API changes, the plugin CI would not fail, but product developer could look at the CI job to see any failing test against master
• (4) we still refresh all plugin CI jobs once a week on Sunday
  • this is current behavior

Summary

• By making those proposed changes, we would meet our Goals above,
• we would also save time and effort as the builds would not fail every week but may only fail after a stable version of Piwik has been released (or when the plugin starts to break against the minimum required piwik).
  • (this may cause Plugin bugs when after a platform stable release, the plugin build fails due to a bug in platform stable release, but this should be a rare case considering that our plugins will only call public APIs and non-deprecated code...)
    • (if we start to have such problems, then we could solve the problems by for example introducing a new routine. Imagine 1 week before stable release, a Product team member or release manager would want to check that the latest beta/RC version works well with our plugins. He would do this by looking at the plugin CI jobs that are allowed failures and would manually check if the failures are due to a cosmetic change or if any failure is due to a regression in core platform.)

what do you think?

may also check external plugins wp-piwik and woocommerce tracking.
e.g. they show data on dashboard which may is broken after update piwik.

@mattab I think this issue is a lot simpler now. PRO builds test against specific Piwik versions only, correct? In this case we only have to maintain the open source plugins that run tests against master.

I think this issue is a lot simpler now. PRO builds test against specific Piwik versions only, correct?

I believe that one suggestion by @mgazdzik and team is to run the PRO builds against the latest Piwik stable version only. @diosmosis I don't know if it's implemented yet though, or whether it's consistently implemented (eg. all premium plugins run builds against latest stable only, while open source plugins run builds against latest stable + master).

Maybe we need to confirm this is what we want to do, and then ensure it's consistently applied across all plugins, then hopefully we will be in a better position with regards to keeping builds green?

we are making some progress

• we are making improvements to CI Status dashboard from time to time in https://github.com/piwik/ci-status/issues
• builds-artifacts now shows a nice and more usable UI: http://builds-artifacts.piwik.org/
• our other ongoing project is to think about, and investigate, remaining UI screenshot tests that fail at random Some UI screenshot tests fail at random  #6693
• @diosmosis is working on making builds green across dozens of plugin repositories, especially after recent DI change and improvements to tests framework in core platform. Epic work 👍

Followed up in https://github.com/piwik/plugins-lifecycle-management/issues/47