CI/CD Processes

AppCat contains many repositories. They all converge in the component-appcat, which is responsible for deploying everything.

In order to streamline our development process, we improved the CI/CD with following steps:

  • Creating a PR in an AppCat repository automatically builds images and packages

  • Creating a PR in an AppCat repository automatically creates a PR on the component that reflects the changes, so it can instantly be deployed any SYN cluster to test

  • Proper labels on the PRs are enforced

  • Merging a PR will automatically create a release, according to the labels it will detect what version increment it needs to do

  • The auto release is not active on the component repository

How to manage the CI/CD pipelines

All the files that specify the process are managed via cruft and cookiecutter. The repository is: github.com/vshn/appcat-cookiecutter. Each of the AppCat repos contains a cronjob that checks for changes every 60 minutes. If there are changes it will open a new PR.

Continuous Integration

There are automated workflows in place in AppCat repository that should streamline our development between AppCat and Component repositories.

Color purple shows user action

Releases from AppCat repository

Example of engineer steps:

  1. Trigger the GitHub Action for a pre-release.

  2. Check and merge the component PR that was opened in step 1.

  3. Tag a new version.

The release has to be manually dispatched, usually before the end of the sprint, on Tuesday. The following is the lifecycle of the release process:

Release Diagram

Feature Requests from AppCat repository

Example of engineer steps:

  1. Create a pull request (PR) with base branch develop.

  2. Label the PR with either patch or minor.

  3. Perform development work.

  4. Approve related PRs in the component and appcat repositories.

  5. Merge the PR by commenting /merge in the appcat repository.

  6. Check the workflow link in the next comment to ensure all jobs succeeded.

  7. Confirm that the develop branch has been updated.

All feature requests starting from AppCat repository have the following lifecycle:

Feature Request Diagram

Hotfix from AppCat

Example of engineer steps:

  1. Create a pull request (PR) with base branch master.

  2. Label the PR with hotfix.

  3. Perform development work.

  4. Approve related PRs in the component/appcat repository.

  5. Merge the PR by commenting /merge in the appcat repository.

  6. Check the workflow link in the next comment to ensure all jobs succeeded.

  7. Confirm that the master branch has been updated.

  8. Confirm PR from master into develop was merged.

  9. Tag a new version.

All hotfixes requests starting from AppCat repository have the following lifecycle:

Hotfix Diagram

Continuous Deployment

AppCat Deployment

Deployment of AppCat is not the same as releasing a new version of the framework to services.

With Crossplane composition functions, multiple versions can be deployed simultaneously in the cluster.

  • These functions run in parallel and remain mostly idle until a composite (service) uses them.

  • This enables a fine-grained release process at the level of each individual instance.

  • Each instance can run any of the available AppCat versions present in the cluster at that time.

A new version is deployed when a new tag is created in the component-appcat repository.

  • In test clusters, deployment happens instantly.

  • In production, it occurs after a one-week delay.

  • The dependency is updated automatically via the commodore-defaults repository using Renovate.

AppCat Rollout

The rollout process refers to a service starting to use a newer composition revision with a higher revision number.

  • The upgrade is triggered automatically during a service maintenance window.

  • The latest available revision is selected during maintenance, even if intermediate versions exist.

  • An upgrade can also be triggered manually by running the maintenance job for a specific instance.

AppCat Rollback

Currently, rollback must be performed manually for each service instance.

  • This involves updating the claim’s spec.compositionRevisionRef.name field to the desired AppCat version.

  • The process is manual.

  • In the near future, Schedar will implement automatic rollback capabilities:

    • Per service instance

    • Per service type

    • Per cluster

Hotfix

Hotfixes are special jobs that update the composition revision reference in composites.

  • A hotfix is triggered for a specific version from the component-appcat repository using the format:

    appcat.deploymentManagementSystem.hotfix=<component-version>-<appcat-version>

  • The specified version must match an existing composition revision.

  • If no match is found, the hotfix jobs will not apply any changes.