I think the latest_stable build should be always an allowed failure, or the dev who adds new features should explicitly make it an allowed failure when making the build green.

The point of testing against latest_stable is to detect when a plugin is currently broken with the latest stable. Seeing red build will help us detect the cases where this plugin's plugin.json does not yet indicate that the plugin requires a more recent Piwik core version. if we can make it automated then we should otherwise we'll forget etc.

Changing a specific entry in matrix/env during generation is very non-trivial (requires correctly & completely parsing travis YAML, detecting build via environment variables + taking into account exclude, correctly outputting formatted YAML w/ preserved comments + other parts). What's in the ticket description may not be possible w/o a lot of work.

Ok makes sense. So if we forget about the issue description, maybe there's something we could do.

For example would it make more sense to rename latest_stable job to be instead minimum_required ? it would run the tests against the minimum required version tag that is specified in the plugin.json, at all times. then we wouldnt need to change the travis.yml generation, and still we would have red build when the minimum required version is not set properly. thoughts?

I think that's a great idea! Should do better than testing against latest stable too.