Kyverno and custom validating admission webhooks as policy engine

Problem

Kubernetes RBAC rules are not suitable to validate certain properties of resources. They can only allow or deny operations of the whole resource. On APPUiO Cloud however, we require fine-grained control of certain properties, such as labels or annotations on Namespaces.

Relevant requirements

Relevant decisions

Proposals

  1. OPA

    Open Policy Agent is a policy engine written for cloud native environments. Policies are written in Rego, which is declarative query language suited for writing policies.

  2. Kyverno

    Kyverno is a Kubernetes native policy engine that uses Kyverno-specific CRDs to configure policies.

  3. kubewarden

    Kubewarden is a Kubernetes native policy engine which allows users to write policies in general purpose languages. Policies are compiled to WebAssembly and loaded into the policy engine as container images.

  4. jsPolicy

    Kubewarden is a Kubernetes native policy engine which allows users to write policies in JavaScript/TypeScript. Interaction with the Kubernetes API is done through the Kubernetes controller-runtime Go library.

  5. Custom validating admission webhook

    It’s possible to write custom validating admission webhooks in Go (or other general purpose languages).

Decision

2022-08-17

  • Custom validating admission webhooks

  • Kyverno

2021-07-30

Kyverno

Rationale

2022-08-17

After a good half year of using Kyverno for all policies in APPUiO Cloud, we’ve come to the conclusion that some of the policies are too complex to be easily maintainable as Kyverno policies. With this realization, we’ve decided to migrate APPUiO Cloud policies to custom validating admission webhooks. Where it makes sense, we’ll migrate policies gradually as we need to make changes to them. We’ll keep Kyverno as an available policy engine for APPUiO Cloud.

We’ve currently decided against using either kubewarden or jsPolicy as a replacement for Kyverno. Kubewarden doesn’t provide all the features required for implementing all APPUiO Cloud policies. In particular, kubewarden currently only supports a limited set of Kubernetes resources (namespaces, ingresses and services) can be queried from kubewarden policies. In contrast, jsPolicy would provide sufficient support to implement all APPUiO Cloud policies. However, using jsPolicy has limited advantages over writing policies directly in Go in a custom validating admission webhook.

This updated decision doesn’t affect the decision on using Kyverno as resource generator.

2021-07-30

Kyverno doesn’t require learning a declarative language compared to OPA. Especially writing mutating policies is subjectively easier in Kyverno than with OPA/Rego.

Choosing Kyverno as policy engine and resource generator reduces complexity.

References