Decision How to Deploy Compositions and XRDs

Problem

We need to deploy XRDs and Compositions for the Application Catalog to Kubernetes clusters using Project Syn, our config management system. Using Project Syn can be seen as a constraint.

Writing the Compositions and Composites ("XRDs") in plain YAML tends to become very verbose and large. Most of the YAML consist of patch operations and many of those are very similar to one another. This makes maintaining and understanding the Compositions very hard. Having helpers with engineering Compositions and XRDs would be a welcomed bonus for the solution.

Requirements

  • Manage XRDs

  • Manage Compositions

  • Manage Providers and their configuration

  • Promote re-usability were possible

  • Help with developing XRDs and Compositions

  • Should support Crossplane packages in the future

  • Should support APPUiO Managed in the future

Proposals

Proposal 1: Commodore Packages With Resources Verbatim

This proposal will enhance the Commodore Component component-crossplane in such a way that it can also deploy XRDs and Compositions via parameters. The definitions for the XRDs and Compositions will live in Commodore Packages. Commodore Packages are used to define XRDs, Compositions and Crossplane Providers in plain YAML (no Jsonnet processing). These packages will be very fine-grained, so there’s a package just to deploy provider-cloudscale, one for provider-exoscale, etc.

They will then be mixed and matched as needed on the various targets. Those targets can be APPUiO Cloud clusters, APPUiO Managed clusters, or simply manifests that could be packaged for general Crossplane usage.

Advantages

  • Verbosity of plain YAML isn’t abstracted again.

Disadvantages

  • Authoring the packages encourages a lot of copy and paste of YAML snippets.

  • The burden of maintaining the resources in plain YAML increases.

deployment1.drawio
Figure 1. Resources deployed via Commodore Packages with plain YAML resources

Proposal 2: Commodore Component With Jsonnet Processing

A new Commodore Component component-appcat will contain parameters and functions to render all the XRDs and Compositions that come in via Commodore Packages. The component will act as a generator for the Compositions and the XRDs. We can use jsonnet functions and snippets to make the generation of the Compositions and XRDs easier. This also promotes re-usability of snippets. The composition exposes parameters to enable and configure the various XRDs and Compositions.

Due to the current lack of conditionals in Crossplane, Jsonnet can be used to create permutations of Compositions for various conditions. Compositions for PostgreSQL with SLA and PostgreSQL without SLA can be generated from the same base.

The parameters in the packages will follow a structure that is optimized for jsonnet generator and the user, therefore the resources get abstracted again.

Advantages

  • Leverages Jsonnet to generate YAML files, reducing a lot of boilerplate YAML.

Disadvantages

  • Complexity of the stack increases.

  • The resources in the packages are abstracted again.

deployment2.drawio
Figure 2. Resources deployed via Commodore Packages and Jsonnet

The main difference between Proposal 1 and 2 is the use of jsonnet in Proposal 2.

The packages can all live in the same, single Git repository (in different files and folders). It’s not required that there is a Git repository for each and every package. However, if using a single Git repo, all resources share the same versions.

Proposal 3: Commodore Component Without Packages

The Commodore Component component-appcat renders all XRDs, Compositions, and Providers. We don’t use any Commodore Packages but write all XRDs and Compositions in Jsonnet. Any configuration will be provided directly as parameters to the component-appcat.

deployment3.drawio
Figure 3. Resources deployed via Commodore Packages and Jsonnet
Advantages

  • Leverages Jsonnet to generate YAML files, reducing a lot of boilerplate YAML.

  • Reduced mental load thanks to fewer layers of abstraction.

  • Commodore Components are a lot more mature than Commodore Packages.

  • We already have experience with writing maintainable Commodore Components.

Disadvantages

  • Complexity of Jsonnet.

  • Less flexibility.

Decision

Proposal 3

Rationale

A Package based solutions have the potential to be more flexible. However, they introduce another layer of abstraction, and it’s hard to reason about multiple interacting packages.

By using an opinionated Commodore Component to deploy all XRDs, Compositions, and Providers, we have a single source for all resources. We can completely leverage the benefits of Jsonnet, reduce boilerplate, and simplify complex configuration by providing a more opinionated interface.

Commodore Packages are not yet well established, while Commodore Components are well understood, mature, and used extensively throughout VSHN,

We deemed the advantages of an opinionated Commodore more important than the lost flexibility.