@sgiehl opened this Pull Request on May 19th 2021 Member

This PR aims to set up a automatic build process.

What this action does:

This action always needs to be triggered manually in the actions tab. This is possible with two purposes:

New release in the current major version

When a new release has to be done this action simply needs to be triggered manually, without setting a version.
The action will then:

  • Check if the user who triggered the action is in the team matomo-org/release-team
    --> If this is not the case the action will abort with an error
  • The version contained in the file core/Version.php will be extracted and used
    --> If a tag with the determined version already exists the action will abort with an error
  • A new tag for that version will be created
  • A new build will be created using the script .github/scripts/build-package.sh
    --> This will create piwik and matomo release archives in zip and tar.gz as well as signature files ending on .asc
  • In addition to this a new release will be created in the repository and the matomo release archives and signatures will be attached. Depending on the version this will be automatically marked as pre-release or not. The release message for each release type can be adjusted if needed. The result can be seen in my fork: Pre-Release | Normal release

Release in LTS version (or re-release of a broken one)

When releasing a LTS version we can't simply trigger the action. Instead we need to create the tag manually first. After this has been done, the actions needs to be triggered while providing the tagged version number in the according input:
image

The action will then:

  • Check if the user who triggered the action is in the team matomo-org/release-team
    --> If this is not the case the action will abort with an error
  • The action now checks if a tag for the given version exists
    --> If no tag exists the action will abort with an error
  • A new build will be created using the script .github/scripts/build-package.sh
    --> This will create piwik and matomo release archives in zip and tar.gz as well as signature files ending on .asc
  • As above a new release in the repo will be created for the tag

Additionally the action could also be used to re-release a "broken" release. Even though this should be an uncommon use case, it's possible to simply trigger the action with a version number that already exists. For pre-releases the script should simply run through and recreate the archives. They should then be replaced in the repo release. For stable releases this currently won't work as the build script aborts if the version is already available on our build server

Requirements:

The team matomo-org/release-team needs to be filled with the persons who should be able to trigger this action. This is currently a private group, so no one can directly see who actually has the permission

The action requires the following repository tokens to be set up:

secret name secret value description
GPG_CERTIFICATE ASCII armored or Base64 encoded GPG certificate that is used to create the signatures for the archives
GPG_CERTIFICATE_PASS Passphrase of the GPG key
RELEASE_PASSWORD Password required to trigger the release action

Testing / Feedback

If someone wants to test the process feel free to get in touch on slack. It's set up on my fork and uses a slightly adjusted build script that uploads the releases to another account. Can give you access...


fixes https://github.com/matomo-org/matomo-package/issues/119

@mattab commented on May 19th 2021 Member

Great to see this progress!

Btw another important step to complete at the end will be to update the instructions at: https://matomo.org/blog/2014/11/verify-signatures-piwik-packages/

@flamisz commented on May 30th 2021 Contributor

Is it ready to review? I see the needs review label but it's still in draft.

@sgiehl commented on May 31st 2021 Member

@flamisz That's because it requires some setup before it could be merged. It's also only a suggestion from my side how the process could work. So it has the needs review and rfc label actually to get some feedback.

@sgiehl commented on June 25th 2021 Member

@mattab @tsteur would be awesome if you can find some time to look into this, so we maybe could set up the automatic release soon.

@mattab commented on June 25th 2021 Member

From my end it looks fantastic @sgiehl - great work! Will save us all time and make our release process faster :rocket:

@tsteur commented on July 20th 2021 Member

@sgiehl what's the status here? can we configure (probably requires @mattab for some keys) and merge and test it?

@sgiehl commented on July 20th 2021 Member

@tsteur Not sure if someone wants to do a code review. Otherwise we can configure it

@tsteur commented on July 20th 2021 Member

👍 wasn't sure as it was set to draft. Generally had a look over it and looked ok-ish but don't know the details. I guess we'd mostly want to test it to ensure it works. Maybe in beginning we wouldn't test it with ssh piwik-builds but some other account.

@github-actions[bot] commented on August 2nd 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@mattab commented on August 2nd 2021 Member

it would be great to work on this soon, as it will help us to introduce another release manager, so we can release more often and regularly.

@github-actions[bot] commented on August 12th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on August 20th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@mattab commented on August 22nd 2021 Member

@sgiehl What are the remaining steps to finish this project? hoping to introduce another release manager, so we can release more often and regularly :rocket:

@sgiehl commented on August 23rd 2021 Member

@mattab I need to find some time looking into or replacing the usage of the external actions we would currently use. Afterwards we need to configure the secrets in GitHub as defined in the PR description create the new GitHub group for release managers and merge this PR.
We could maybe first test it with a SSH key that actually uploads the packages somewhere else.

@github-actions[bot] commented on August 31st 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on September 8th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on September 16th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@mattab commented on September 22nd 2021 Member

@sgiehl Just curious (it's not urgent), when do you think you could work on this project (in how many weeks)? (after your current priorities of Hits + ongoing PRs reviews)

@github-actions[bot] commented on September 30th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on October 14th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on October 28th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on November 5th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on November 13th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@sgiehl commented on November 24th 2021 Member

I've started to resume the work on that one and got some questions/notes about the future process I would like to clarify.

  • Currently we are using the build_package.sh in the other repo to build releases. To reduce the complexity a bit I have moved it into the .github folder in this repo. That way I can also adjust it a bit more to match what is really needed for the action
  • Currently the build script uses a ssh key to connect to our build server in order to upload the releases. This works smoothly. Nevertheless I was wondering if we actually want to store a valid ssh key that allows connecting to our server as a github secret.
    An alternative would be to create a script on our server that can be triggered via url, which would then download the release packages attached to the github release and store it on the server. That way the risk of misuse would be lowered, but it will for sure take some extra effort to develop it.
  • Also if you have any wishes how to change / improve the build process (described in the PR description) please let me know.

@tsteur @mattab @justinvelluppillai your feedback would be welcome

@tsteur commented on November 24th 2021 Member

An alternative would be to create a script on our server that can be triggered via url, which would then download the release packages attached to the github release and store it on the server. That way the risk of misuse would be lowered, but it will for sure take some extra effort to develop it.

I guess it might be fairly easily developed? We would also want to have a very secure password there. It then gets quickly complicated as you'd want technically then brute force prevention etc but if it only downloads the script from Github then it may be fine and not needed as much.

The risk is that someone could trigger a release of some random branch etc and then get the server to download it. But at least it wouldn't allow to change the latest version number and not any other packages assuming it's all written correctly. This risk exists though with either solution. Does the github action already enforce that we only allow a release RC and production releases from next_release and beta releases additionally from 4.x-dev? I've now just set up a branch protection rule for next_release so there's no force push possible. I could also require "PRs" and approvals of at least one person for next_release. It's not done yet, but could do (not done yet)
image

The zip is only one thing thought right? There's also the version number that needs to be updated on api.matomo.org?

Would it overall maybe easier to simply/better setting up a webhook? We can define a secret etc. I haven't thought too much about it though.I suppose we want to attach the release to the github release but maybe that's not needed? Just some thoughts... There are probably some downsides to it. Mostly that you then want some harder brute force protection maybe etc. Overall might be more complicated.

@tsteur commented on November 24th 2021 Member

would have been nice if github allowed configuring creating releases only from specific branches. doesn't seem to be the case if I see this right

@sgiehl commented on November 25th 2021 Member

I'll leave it as is for now, so the upload will be done using an ssh key. We can also start with that approach and improve/harden it when it's already running. Will now check for other improvements to finish the PR.

Does the github action already enforce that we only allow a release RC and production releases from next_release and beta releases additionally from 4.x-dev?

No that is not enforced yet. It's possible to trigger the action for a any branch (where the action exists). On that branch a tag will be created. But I can add some checks, so this is only possible for next_release or 4.x-dev. Btw. it will still be possible to create a tag manually on any branch and trigger the action providing that tag to build it. (This will be useful for building LTS version or if we ever need to release a fix for an older version or rebuilt one.)

I've now just set up a branch protection rule for next_release so there's no force push possible. I could also require "PRs" and approvals of at least one person for next_release. It's not done yet, but could do (not done yet)

That would make sense to me.

I suppose we want to attach the release to the github release but maybe that's not needed?

I guess we hadn't done that before, as it would have been extra effort to do that. But with a github action it can be done automatically, so I can't see any downside in doing that. Besides the fact that we no longer might have proper download stats (unless a github webhook is also triggered for downloads)

@tsteur commented on November 28th 2021 Member

No that is not enforced yet. It's possible to trigger the action for a any branch (where the action exists). On that branch a tag will be created. But I can add some checks, so this is only possible for next_release or 4.x-dev. Btw. it will still be possible to create a tag manually on any branch and trigger the action providing that tag to build it. (This will be useful for building LTS version or if we ever need to release a fix for an older version or rebuilt one.)

For security it would be great to only allow \d.x-dev and next_release for this for security as these branches are protected. Technically could even only allow next_release for even better security and require a comment or someone to confirm the PR.

Overall downloading the release from github might be then more secure then using SSH I reckon. As in worst case nobody can change historical releases and zips etc.

@mattab commented on November 29th 2021 Member

+1 for create a script on our server that can be triggered via url, which would then download the release packages attached to the github release and store it on the server. That way the risk of misuse would be lowered and any other "safeguards" we can put in to protect releases

@github-actions[bot] commented on December 6th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on December 14th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on December 22nd 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on December 30th 2021 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on January 6th 2022 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@justinvelluppillai commented on January 6th 2022 Contributor

@sgiehl be good to prioritise this to see if we can use it to release 4.7.0 soon. Is there anything you're waiting on to proceed?

@github-actions[bot] commented on January 14th 2022 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on January 26th 2022 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@github-actions[bot] commented on February 3rd 2022 Contributor

This issue is in "needs review" but there has been no activity for 7 days. ping @matomo-org/core-reviewers

@sgiehl commented on February 3rd 2022 Member

Just to give an update on the current state:

The build script is currently working as described in the PR description. Theoretically it could be set up and it would already be possible to use it for building releases.

I've today managed to replace the use of the action crazy-max/ghaction-import-gpg with custom commands.

Last point would be to replace the mechanism to upload the releases to the build server with a server side script that is triggered from the action. Not yet sure how to solve that, will check tomorrow.

@justinvelluppillai commented on February 3rd 2022 Contributor

Thanks for the update @sgiehl. It would be good to do a test run of this, could we try with a beta release later this month?

@sgiehl commented on February 4th 2022 Member

@mattab @tsteur Is it required to change the mechanism from uploading files to the server(s) to triggering a script on the server(s) that downloads the builds instead?

It's a way more complex than I initially thought. Currently we are uploading the releases as matomo-X.zip and piwik-X.zip to the server. Depending on the released version the version files like LATEST_STABLE, LATEST_BETA, ... are updated as well. That is not only done on the builds server, but the latter files are updated on API and WWW as well.
Looking at the code it seems at least the update on WWW can't work. It's done here, but actually $WWW_PATH is not defined and so it would try to upload to /, which I guess won't work.
Also I wonder if the certain updates to the API server are needed or if wouldn't simply create all those version related files on the build server and let the API server access those.
If not we would either need to have a separate script on the API server that is triggered while releasing or we would need to ssh from the build server to the api server to perform the updates that way.

Also I personally would only attach matomo.zip (and .tar.gz) to the github releases, as I don't see any advantage in also attaching a piwik.zip there. But if we download the release files from github, we would need to attach it or decide to not provide piwik.zip on the release server anymore (guess this might cause problems when updating from an older version).

Any thoughts?

@tsteur commented on February 8th 2022 Member

@sgiehl I'm not sure I fully understand re the problem. Generally whether it is piwik.zip or matomo.zip shouldn't really matter as we can just create the needed files on demand or am I missing something? Whatever script downloads these files could just create the needed file names and copy it over?

Re www path and API: we could use an API endpoint there as well to update these files or still use ssh there.

I'm thinking for WWW_PATH: We could refactor matomo.org to simply get the right data from builds.matomo.org. This should be a very trivial change of just changing the file path/ url (but didn't look). Or it could be a cron that syncs it with builds.matomo.org regularly like wget https://builds.matomo.org/LATEST...

Maybe on api.matomo.org we could have a cronjob that updates local LATEST_ files as well and copies them over from builds.matomo.org? We would just need to make sure that in future we also have all the entries like LATEST_2X and LATEST_2x_BETA etc on builds.matomo.org as well.

Not sure if there be any issue with that? Maybe changing matomo.org and the api like this generally be a good idea to have only one source of truth and to not needing SSH access to 3 systems? Instead it be only one.

As for builds.matomo.org: We could use an ssh key + secure password or an HTTP endpoint that will get required information and download / create the needed files. Maybe SSH is more secure assuming that we can prevent brute force attacks etc with fail2ban or similar software vs on an endpoint we need to implement things like brute force protection ourselves. Technically, even if someone did brute force the webhook endpoint for example, then they still wouldn't be able to do much as long as they can't also trigger a release. Vs if someone had the SSH keys for some reason, then it be an issue. So thinking some endpoint might be better?

@sgiehl commented on February 8th 2022 Member

@tsteur the difference between piwik.zip and matomo.zip is not only the filename. It contains a folder with the same name. And iirc when Matomo is updating it looks for a folder named piwik or matomo in the downloaded release. Older versions of Piwik might though only look for a folder named piwik, which might then cause problems if that isn't the case.

Creating all version-related info files only on the builds server sounds good. Guess that makes everything a bit more stable. So we would need to adjust API and WWW to use those files instead (or sync them somehow). I guess I don't have access to those repos (anymore). Can you give me access, I would then check where they are actually used, so we can maybe also clean that up a bit and avoid creating unused files in the release process.

Regarding SSH connection VS HTTP Endpoint:

The script currently already does a SSH connect to upload the build files and adjust the version files. Currently this is done without a ssh key password. Not sure if there is a big benefit in using a password, as we would need to provide the ssh key and the password as github secrets. So if someone gains access to such secrets somehow, there is no big difference. But it should be possible to use a password for sure.

Using an HTTP endpoint would be an alternative way. Writing a simple PHP script, that receives a version number that should be downloaded / updated is quite easy. But as you mentioned it requires some more work around that to set it up properly.

Personally I don't have a big preference, but the HTTP endpoint might be more secure. We could also decide to go with SSH for now and think about building and using an HTTP endpoint instead later.

@tsteur commented on February 9th 2022 Member

For WWW I could just set up two cronjobs like cd /home/... && curl -O https://builds.matomo.org/LATEST (also for LATEST_SIZE).

For API: You have access now: https://github.com/matomo-org/api.matomo.org/settings/access?guidance_task= Maybe easiest be to have a small php script that we can run in a cronjob every 5 minutes and downloads all LATEST* files from build server. I could set up the cron.

Remains the issue with piwik.zip vs matomo.zip. The only solution would be to attach both release files in that case?

Re SSH it be better to have a password on it just to be more safe in case token gets leaked. I would maybe directly build the HTTP solution though as it might be not too much work. If we don't want to have a public API we could also just have a script that checks every 5 minutes for a new release? Then we won't need to worry about passwords and what not and no files be directly accessible on the server.

@sgiehl commented on February 9th 2022 Member

For WWW I could just set up two cronjobs like cd /home/... && curl -O https://builds.matomo.org/LATEST (also for LATEST_SIZE).

That sounds fine.

For API: You have access now: https://github.com/matomo-org/api.matomo.org/settings/access?guidance_task= Maybe easiest be to have a small php script that we can run in a cronjob every 5 minutes and downloads all LATEST* files from build server. I could set up the cron.

I'll prepare a simple script to fetch the files from builds.matomo.org and store them locally. Will create a PR once it's done.

Remains the issue with piwik.zip vs matomo.zip. The only solution would be to attach both release files in that case?

If we download the github assets that's the only solution I can think of. Otherwise we would need to kind of rebuilt the package on the build server after downloading it.
Other solutions would be to either discontinue updating the piwik. files (So they will stay on the current version. That would at least allow older versions to update to that version and from there to the latest version using matomo.) or to simply symlink them to matomo.* files (that might break automatic updates from version before the rebrand, but maybe thats a tolerable thing, as manual update would still work though).

Re SSH it be better to have a password on it just to be more safe in case token gets leaked. I would maybe directly build the HTTP solution though as it might be not too much work. If we don't want to have a public API we could also just have a script that checks every 5 minutes for a new release? Then we won't need to worry about passwords and what not and no files be directly accessible on the server.

I have started to build a script, that automatically detects if a new release is available on github and downloads / updates the required files. Not sure if we should run it every 5 minutes, as it might hit the github API limit without authentication.
Should I create a new private repo for that single file, or shall we include it somewhere else?

@tsteur commented on February 9th 2022 Member

We can use a new repository for this 👍 We can also configure a token to not run into quota limits.

I wouldn't build the zip files again on the build server, there would be issues over time. I don't think it's a big issue to have piwik.zip and matomo.zip attached to the release. Most people will download it on the website or directly or through Matomo and only few people will download it from there and it shouldn't be a problem if both files are attached.

@sgiehl commented on February 10th 2022 Member

I have cleaned up the action now, so it doesn't do any remote work and also attaches the piwik files.
The part here should be fully ready to review and for testing.
@justinvelluppillai @tsteur @mattab If someone of you wants to have a quick demo of that, let me know and we can setup a quick video call for that. I can easily demonstrate that on my fork.

I'm still working on finishing the scripts that will be running on the builds and api server. But they can later be checked and reviewed separately

@justinvelluppillai commented on February 14th 2022 Contributor

Thanks @sgiehl I will see if I can set up a call for demo asap.

@sgiehl commented on February 18th 2022 Member

@Findus23 applied some of the fixes

@tsteur commented on February 22nd 2022 Member

👍

@sgiehl commented on February 23rd 2022 Member

Guess we actually could merge this one now. Once the secrets have been added we can give it a first try...

This Pull Request was closed on February 24th 2022
Powered by GitHub Issue Mirror