@sgiehl opened this Pull Request on May 19th 2021 Member

This PR aims to set up a automatic build process.

What this action does:

This action always needs to be triggered manually in the actions tab. This is possible with two purposes:

New release in the current major version

When a new release has to be done this action simply needs to be triggered manually, without setting a version.
The action will then:

  • Check if the user who triggered the action is in the team matomo-org/release-team (needs to be created)
    --> If this is not the case the action will abort with an error
  • The version contained in the file core/Version.php will be extracted and used
    --> If a tag with the determined version already exists the action will abort with an error
  • A new tag for that version will be created
  • The default branch of matomo-org/matomo-package will be checked out and scripts/build-package.sh will be executed
    --> This will create piwik and matomo release archives in zip and tar.gz as well as signature files ending on .asc and upload them to the matomo server
  • In addition to this a new release will be created in the repository and the matomo release archives and signatures will be attached. Depending on the version this will be automatically marked as pre-release or not. The release message for each release type can be adjusted if needed. The result can be seen in my fork: Pre-Release | Normal release

Release in LTS version (or re-release of a broken one)

When releasing a LTS version we can't simply trigger the action. Instead we need to create the tag manually first. After this has been done, the actions needs to be triggered while providing the tagged version number in the according input:
image

The action will then:

  • Check if the user who triggered the action is in the team matomo-org/release-team (needs to be created)
    --> If this is not the case the action will abort with an error
  • The action now checks if a tag for the given version exists
    --> If no tag exists the action will abort with an error
  • The default branch of matomo-org/matomo-package will be checked out and scripts/build-package.sh will be executed
    --> This will create piwik and matomo release archives in zip and tar.gz as well as signature files ending on .asc and upload them to the matomo server
  • As above a new release in the repo will be created for the tag

Additionally the action could also be used to re-release a "broken" release. Even though this should be an uncommon use case, it's possible to simply trigger the action with a version number that already exists. For pre-releases the script should simply run through and recreate the archives. They should then be replaced on the server and in the repo release. For stable releases this currently won't work as the build script aborts if the version is already available on our build server

Requirements:

The team matomo-org/release-team needs to be created and filled with the persons who should be able to trigger this action. This can also be a private group, so no one can directly see who actually has the permission

The action requires the following repository tokens to be set up:

secret name secret value description
GITTOKEN A GitHub API token that is used to check if the current actor is within the release group (e.g. the token's users must be in this group if it's not public)
GPG_CERTIFICATE ASCII armored or Base64 encoded GPG certificate that is used to create the signatures for the archives
GPG_CERTIFICATE_PASS Passphrase of the GPG key
SSH_KEY SSH private key that will be used to connect to the Matomo server (the key shouldn't be password protected)

Testing / Feedback

If someone wants to test the process feel free to get in touch on slack. It's set up on my fork and uses a slightly adjusted build script that uploads the releases to another account. Can give you access...


fixes https://github.com/matomo-org/matomo-package/issues/119

@mattab commented on May 19th 2021 Member

Great to see this progress!

Btw another important step to complete at the end will be to update the instructions at: https://matomo.org/blog/2014/11/verify-signatures-piwik-packages/

@flamisz commented on May 30th 2021 Contributor

Is it ready to review? I see the needs review label but it's still in draft.

@sgiehl commented on May 31st 2021 Member

@flamisz That's because it requires some setup before it could be merged. It's also only a suggestion from my side how the process could work. So it has the needs review and rfc label actually to get some feedback.

@sgiehl commented on June 25th 2021 Member

@mattab @tsteur would be awesome if you can find some time to look into this, so we maybe could set up the automatic release soon.

@mattab commented on June 25th 2021 Member

From my end it looks fantastic @sgiehl - great work! Will save us all time and make our release process faster :rocket:

@tsteur commented on July 20th 2021 Member

@sgiehl what's the status here? can we configure (probably requires @mattab for some keys) and merge and test it?

@sgiehl commented on July 20th 2021 Member

@tsteur Not sure if someone wants to do a code review. Otherwise we can configure it

@tsteur commented on July 20th 2021 Member

👍 wasn't sure as it was set to draft. Generally had a look over it and looked ok-ish but don't know the details. I guess we'd mostly want to test it to ensure it works. Maybe in beginning we wouldn't test it with ssh piwik-builds but some other account.

@github-actions[bot] commented on August 2nd 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@mattab commented on August 2nd 2021 Member

it would be great to work on this soon, as it will help us to introduce another release manager, so we can release more often and regularly.

@github-actions[bot] commented on August 12th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on August 20th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@mattab commented on August 22nd 2021 Member

@sgiehl What are the remaining steps to finish this project? hoping to introduce another release manager, so we can release more often and regularly :rocket:

@sgiehl commented on August 23rd 2021 Member

@mattab I need to find some time looking into or replacing the usage of the external actions we would currently use. Afterwards we need to configure the secrets in GitHub as defined in the PR description create the new GitHub group for release managers and merge this PR.
We could maybe first test it with a SSH key that actually uploads the packages somewhere else.

@github-actions[bot] commented on August 31st 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on September 8th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on September 16th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@mattab commented on September 22nd 2021 Member

@sgiehl Just curious (it's not urgent), when do you think you could work on this project (in how many weeks)? (after your current priorities of Hits + ongoing PRs reviews)

@github-actions[bot] commented on September 30th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on October 14th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on October 28th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on November 5th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on November 13th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@sgiehl commented on November 24th 2021 Member

I've started to resume the work on that one and got some questions/notes about the future process I would like to clarify.

  • Currently we are using the build_package.sh in the other repo to build releases. To reduce the complexity a bit I have moved it into the .github folder in this repo. That way I can also adjust it a bit more to match what is really needed for the action
  • Currently the build script uses a ssh key to connect to our build server in order to upload the releases. This works smoothly. Nevertheless I was wondering if we actually want to store a valid ssh key that allows connecting to our server as a github secret.
    An alternative would be to create a script on our server that can be triggered via url, which would then download the release packages attached to the github release and store it on the server. That way the risk of misuse would be lowered, but it will for sure take some extra effort to develop it.
  • Also if you have any wishes how to change / improve the build process (described in the PR description) please let me know.

@tsteur @mattab @justinvelluppillai your feedback would be welcome

@tsteur commented on November 24th 2021 Member

An alternative would be to create a script on our server that can be triggered via url, which would then download the release packages attached to the github release and store it on the server. That way the risk of misuse would be lowered, but it will for sure take some extra effort to develop it.

I guess it might be fairly easily developed? We would also want to have a very secure password there. It then gets quickly complicated as you'd want technically then brute force prevention etc but if it only downloads the script from Github then it may be fine and not needed as much.

The risk is that someone could trigger a release of some random branch etc and then get the server to download it. But at least it wouldn't allow to change the latest version number and not any other packages assuming it's all written correctly. This risk exists though with either solution. Does the github action already enforce that we only allow a release RC and production releases from next_release and beta releases additionally from 4.x-dev? I've now just set up a branch protection rule for next_release so there's no force push possible. I could also require "PRs" and approvals of at least one person for next_release. It's not done yet, but could do (not done yet)
image

The zip is only one thing thought right? There's also the version number that needs to be updated on api.matomo.org?

Would it overall maybe easier to simply/better setting up a webhook? We can define a secret etc. I haven't thought too much about it though.I suppose we want to attach the release to the github release but maybe that's not needed? Just some thoughts... There are probably some downsides to it. Mostly that you then want some harder brute force protection maybe etc. Overall might be more complicated.

@tsteur commented on November 24th 2021 Member

would have been nice if github allowed configuring creating releases only from specific branches. doesn't seem to be the case if I see this right

@sgiehl commented on November 25th 2021 Member

I'll leave it as is for now, so the upload will be done using an ssh key. We can also start with that approach and improve/harden it when it's already running. Will now check for other improvements to finish the PR.

Does the github action already enforce that we only allow a release RC and production releases from next_release and beta releases additionally from 4.x-dev?

No that is not enforced yet. It's possible to trigger the action for a any branch (where the action exists). On that branch a tag will be created. But I can add some checks, so this is only possible for next_release or 4.x-dev. Btw. it will still be possible to create a tag manually on any branch and trigger the action providing that tag to build it. (This will be useful for building LTS version or if we ever need to release a fix for an older version or rebuilt one.)

I've now just set up a branch protection rule for next_release so there's no force push possible. I could also require "PRs" and approvals of at least one person for next_release. It's not done yet, but could do (not done yet)

That would make sense to me.

I suppose we want to attach the release to the github release but maybe that's not needed?

I guess we hadn't done that before, as it would have been extra effort to do that. But with a github action it can be done automatically, so I can't see any downside in doing that. Besides the fact that we no longer might have proper download stats (unless a github webhook is also triggered for downloads)

@tsteur commented on November 28th 2021 Member

No that is not enforced yet. It's possible to trigger the action for a any branch (where the action exists). On that branch a tag will be created. But I can add some checks, so this is only possible for next_release or 4.x-dev. Btw. it will still be possible to create a tag manually on any branch and trigger the action providing that tag to build it. (This will be useful for building LTS version or if we ever need to release a fix for an older version or rebuilt one.)

For security it would be great to only allow \d.x-dev and next_release for this for security as these branches are protected. Technically could even only allow next_release for even better security and require a comment or someone to confirm the PR.

Overall downloading the release from github might be then more secure then using SSH I reckon. As in worst case nobody can change historical releases and zips etc.

@mattab commented on November 29th 2021 Member

+1 for create a script on our server that can be triggered via url, which would then download the release packages attached to the github release and store it on the server. That way the risk of misuse would be lowered and any other "safeguards" we can put in to protect releases

Powered by GitHub Issue Mirror