Honestly I think this would just delay the issues a few days as most people would then use this new releases channels and therefore most bugs would again only be found after the update migrates to this channel. In addition, I think this would cause a bit confusion for users.

I'd propose the opposite direction: Make the release candidates more well-known.
I think we should make it more well-known that an easy way to contribute to Matomo without needing any technical knowledge, is to install the release candidates of Matomo on a copy of their Matomo instance in the weeks before a new Matomo release and report any issues that occur there.
That way the most severe issues can be fixed before the new release.

I think if you want it extra stable, the best practice is really to ideally wait for a week or two before updating to the next version. For some software I wait for 1-3 months before updating. It's of course bit annoying when there are security issues, but usually the fixed security issues in the last years have been pretty much minor.

I personally do not update anything for quite some time, but there are A LOT of users doing otherwise. I have no strong opinion on this one but I think just using RCs would not have found that archiving error as there would probally be no visits to archive on a copied instance.

I think some users do upgrade to RC on their production instance. Not sure how many RC testers there were and whether there was enough time between merging PR that caused the issue and the release. Maybe that's an issue to just really wait for a week after RC before creating the release.(we might have mentioned this before and would probably need to adjust processes to ensure that) not sure if that was the problem though.

Perhaps we can keep track of how many users upgrade to a RC so we can be better aware of how battle ready a potential release is.

The problem that I realised again the past two days are, that a lot of people update on the first day a release is there and a big number of them are just using Matomo as users and have no clue about what could be the issue (I looked like 10 minutes on the PRs and was like 90% confident that I found the correct PR). So there has to be a way to protect the users against a situation like this, or the reputation of Matomo will shift more to "Never update, it could break everything".

Ideally we'd do this by releasing software w/o issues the first time around. 2 of the 3 issues caught in this release might have been caught by the OneClickUpdate test which hasn't been merged yet. The other issue points to two blind spots in our testing (LDAP + emails). And releasing software w/ these issues points to a lack of knowledge of how stable a release will be and exactly how users run/update/configure matomo. If we can address root cause issues like these, then we can release better software and wouldn't need a new release channel. They just need to be made a priority.

Amen

2 of the 3 issues caught in this release might have been caught by the OneClickUpdate test which hasn't been merged yet. The other issue points to two blind spots in our testing (LDAP + emails).

IMHO automated testing is the only way to get stable releases. One test is on the way. So to improve release stability, writing tests especially for the blind spots is essential. Introducing new release channels (extra stable, super stable, super extra stable, ...) is the wrong way and not the mindset of OSS.

I'm using matomo aka piwik for a couple of years and the release quality is compared to other OSS projects pretty high. Thank you matomo-team for that.

👍 @jgkirschbaum

I'm thinking most releases go without problems and it is unrealistic to think you can avoid all problems. We're already focusing a LOT on tests and adding more tests with every release etc. There's the beta/RC releases also to catch errors and of course we have to rely on the community to help testing. We're also maintaining a few Matomo's ourselves where we test the update process. I think we can maybe do better in terms of having a feature freeze, release an RC, do nothing for a week, and hope to get feedback if there is any problem. So far we sometimes still merge things after RC or close before the release which shouldn't happen.

I think we can close this issue?

Thanks everyone for feedback. We'll continue doing our best to release stable software. Sometimes there will be a bug or two sneaking in, but we have thousands of automated tests, and most pull requests also come with their own new tests, so we're always making this better. In the future we're hoping to release more often (ideally once a month) which will also help make more stable releases eventually. 🚀