Kyverno and custom validating admission webhooks as policy engine
Kubernetes RBAC rules are not suitable to validate certain properties of resources. They can only allow or deny operations of the whole resource. On APPUiO Cloud however, we require fine-grained control of certain properties, such as labels or annotations on Namespaces.
Open Policy Agent is a policy engine written for cloud native environments. Policies are written in Rego, which is declarative query language suited for writing policies.
Kyverno is a Kubernetes native policy engine that uses Kyverno-specific CRDs to configure policies.
Kubewarden is a Kubernetes native policy engine which allows users to write policies in general purpose languages. Policies are compiled to WebAssembly and loaded into the policy engine as container images.
It’s possible to write custom validating admission webhooks in Go (or other general purpose languages).
After a good half year of using Kyverno for all policies in APPUiO Cloud, we’ve come to the conclusion that some of the policies are too complex to be easily maintainable as Kyverno policies. With this realization, we’ve decided to migrate APPUiO Cloud policies to custom validating admission webhooks. Where it makes sense, we’ll migrate policies gradually as we need to make changes to them. We’ll keep Kyverno as an available policy engine for APPUiO Cloud.
We’ve currently decided against using either kubewarden or jsPolicy as a replacement for Kyverno. Kubewarden doesn’t provide all the features required for implementing all APPUiO Cloud policies. In particular, kubewarden currently only supports a limited set of Kubernetes resources (namespaces, ingresses and services) can be queried from kubewarden policies. In contrast, jsPolicy would provide sufficient support to implement all APPUiO Cloud policies. However, using jsPolicy has limited advantages over writing policies directly in Go in a custom validating admission webhook.
|This updated decision doesn’t affect the decision on using Kyverno as resource generator.|