Protobuf

Protobuf, short for Protocol Buffers, is a language-agnostic binary serialization format developed by Google. It is designed to be a compact, efficient, and extensible way of serializing structured data. Protobuf allows you to define your data structures and generate code in multiple programming languages to handle serialization and deserialization.

In the context of Kubernetes, Protobuf is used as the serialization format for the API communication between various components of the system. Kubernetes defines its API using Protobuf, and the Protobuf definition files are used to generate the client libraries and server stubs for different programming languages. For more information checkout the official documentation from kubernetes.

Protobuf in API Server

The Kubernetes API server supports three data formats for resource specifications: JSON, YAML, and protobuf. Protobuf is the most efficient format, but it is also the most complex. During development of API Servers in VSHN it was noticed that cluster scoped resources do not require protobuf data format to be configured. The story is different for namespaced resources. When a namespace is being deleted the namespace scoped resources must implement protobuf in order for cascading deletion to work. This phenomena may also apply to cluster scoped resources when they are dependants on other resources.

Add protobuf support for a Kubernetes API server

The following is a guide on how to add protobuf support for your APIs

Prerequisites

  • The APIs are already implemented for the API server.

The protobuf compiler ingests API definitions and outputs go code that is able to serialize and deserialize protobuf message.

Procedure

  1. Install necessary plugins and dependencies in your API server repository. Make sure you get the latest version.

    # necessary dependency
go install golang.org/x/tools/cmd/goimports@latest

# kubernetes source code is necessary for the autogenerated code
git clone -q --depth 1 https://github.com/kubernetes/kubernetes.git <tmp>/kubernetes

# necessary dependency
touch boilerplate.txt

# protoc-gen-gogo plugin is used to generate the final go code for our APIs
go install github.com/gogo/protobuf/protoc-gen-gogo@latest

# protoc main plugin used by the protoc-gen-gogo
wget -q -O protoc.zip https://github.com/protocolbuffers/protobuf/releases/download/v22.1/protoc-22.1-linux-x86_64.zip
unzip protoc.zip -d <your-bin-folder>
rm protoc.zip

  2. Run protobuf code generator

    go run k8s.io/code-generator/cmd/go-to-protobuf \
    --packages=<package-to-you-apis> \
    --output-base=./<tmp> \
    --go-header-file=./boilerplate.txt  \
    --apimachinery-packages='-k8s.io/apimachinery/pkg/util/intstr,-k8s.io/apimachinery/pkg/api/resource,-k8s.io/apimachinery/pkg/runtime/schema,-k8s.io/apimachinery/pkg/runtime,-k8s.io/apimachinery/pkg/apis/meta/v1,-k8s.io/apimachinery/pkg/apis/meta/v1beta1,-k8s.io/api/core/v1,-k8s.io/api/rbac/v1' \
    --proto-import=./<tmp>/kubernetes/vendor/
The package should contain the full name with path ex. github.com/vshn/appcat/apis/appcat/v1