ADR0012 - Testing Concept

One of the biggest issues with working with Operators and CRDs in K8s is that they are singletons. While there are ways to make breaking changes seamless(er), it’s too much effort for simple tests.

Chrysopoeia generates distinct API groups based on the name of the CustomResourceDefinitionSource. Chainsaw tests in the reagent (service chart) repositories leverage this behaviour by adding an ID part to the generated sources. That ID is the `values.yaml’s content hashed together with the claim namespace. Since Chainsaw generates a random namespace for each of the tests, every single e2e test run will have its own set of CRDs to work with. This is even true when Chainsaw runs tests for the same version in parallel in various namespaces.

There are multiple layers to testing and each one needs one or multiple tools to help with it.

Chainsaw is the more powerful successor to Kuttl, which was previously used. It fixes most of the pain points of Kuttl which makes it a very comfortable tool to do e2e tests with K8s.

envtest is the de-facto integration test tooling for kubebuilder projects. It helps spin up enough K8s to test parts of a controller.

Helm unittests provide unit testing for Helm charts. The concept is similar to Chainsaw and Kuttl. It renders the chart and provides assertion and matching rules to check the output against.

Reagents bring a default CI setup which triggers a default Chainsaw test on each PR. The default Chainsaw test works out of the box and simply checks if the Helm chart installs or not. A service maintainer can then add service-specific asserts or even whole new steps to test advanced topics, like scaling, backup or restore. Either unittests or golden tests should be maintained for each service.

Controllers should have integration and unit tests. These should run on every PR via CI and on the release CI as well. Chainsaw e2e tests should be added to controllers if sensible.