Setting up the Crossplane Service Broker

This tutorial is deprecated. Some links may not work.

This is a tutorial on how to install the Crossplane Service Broker as a service in your cluster and how to use Bearer Tokens to authenticate the Service Catalog against that Crossplane Service Broker service.

This tutorial assumes some basic knowledge about how a Service Catalog and a (generic) Service Broker work together. To learn more about the whole Service Catalog / Service Broker concept, read the relevant Kubernetes documentation, which is excellent.

Overview

The setup around a Crossplane Service Broker consists of three main components:

The Service Catalog is the entry-point for users who want to instantiate new services. Its interface is a well-defined REST-API which provides an overview over all available and all instantiated services. It connects to one (or many) Crossplane Service Brokers, which provide this information back to the Service Catalog. (The Service Catalog may also connect to other kinds of Service Brokers in order to provide more services.)

The Service Catalog has been deprecated.
A Crossplane Service Broker that knows which services it offers and what already instantiated services it’s responsible for. When a user wants to instantiate a certain service (via the Service Catalog), the Crossplane Service Broker creates the respective Crossplane custom resources in the Kubernetes cluster in which it runs.
Crossplane will then react on those Crossplane custom resources and from then on manage the lifecycle of the actual service instance, for example Redis.

The Service Catalog and the Crossplane Service Broker don’t necessarily have to run on the same Kubernetes cluster. For example, a Service Catalog can be installed on the Kubernetes cluster of a customer, while the Crossplane Service Broker might be running on a central and dedicated cluster.

Additionally, services managed by Crossplane may be provisioned outside of the cluster on which Crossplane is installed on. For example, Crossplane supports instantiating cloud resources from providers such as Google Cloud, Azure or AWS. Naturally, such resources don’t run on the cluster on which Crossplane is installed.

Preparations

This tutorial assumes that you have some resources and services already at your disposal and already configured:

Have at least one Kubernetes cluster with Crossplane installed.
Know the service id of at least one service which Crossplane manages.
Have access to an authentication server which issues Bearer Tokens and which has a JWKS endpoint, like Keycloak for example.
Have svcat installed on your computer.

If you don’t have a Crossplane setup with services configured, then, in order to follow along this guide, have a look at vshn/application-catalog-demo: It explains how to setup Crossplane and how to prepare services that are later recognized by the Crossplane Service Broker.

Note that the service id used in that demo is redis-k8s. Use it instead of the UUIDs when defining OSB_SERVICE_IDS in the deployment.yaml.

Furthermore, you may also want to have a look at the Implement a New Service how-to. It explains the intricate details of how a service must be defined such that the Crossplane Service Broker can make use of it.

Install the Service Catalog

The command below will install the service catalog service in your current Kubernetes cluster using Helm in the catalog namespace. The namespace will be created if it doesn’t already exist.

Before you can run the command below, you need to create a file called values.yaml with the following content:

values.yaml

image: quay.io/kubernetes-service-catalog/service-catalog:v0.3.1 (1)
securityContext: { runAsUser: 1001 } (2)

1	The specific Docker image of the service catalog that will be deployed.
2	Configuring the service catalog such that it doesn’t run as root. This may not be necessary in your cluster.

Installation

Run the following command to install the service catalog:

helm install \
  catalog \ (1)
  --values values.yaml \ (2)
  --namespace catalog \ (3)
  --create-namespace \ (4)
  --repo https://kubernetes-sigs.github.io/service-catalog \ (5)
  catalog (6)

1	The name of the service in your Kubernetes cluster.
2	Read configuration options from `values.yaml` file.
3	The namespace in which the service catalog will be installed into.
4	The previously defined namespace will be created if it doesn’t yet exist.
5	The Helm repository that contains the chart for the service catalog.
6	The name of the chart which Helm shall apply.

Install the Crossplane Service Broker

The Crossplane Service Broker is configured through environment variables. A typical deployment may look like the following YAML.

After you created the file deployment.yaml below, run this command to install the Crossplane Service Broker in your Kubernetes cluster:

kubectl apply -f deployment.yaml

deployment.yaml

kind: Namespace (1)
apiVersion: v1
metadata:
  labels:
    name: service-broker
  name: service-broker
---
kind: Deployment (2)
apiVersion: apps/v1
metadata:
  name: service-broker-test (3)
  namespace: service-broker
  labels:
    name: service-broker-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/instance: test
      app.kubernetes.io/name: service-broker
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: test
        app.kubernetes.io/name: service-broker
        name: service-broker-test
    spec:
      containers:
        - name: service-broker
          image: quay.io/vshn/crossplane-service-broker:v0.4.1
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          env: (4)
            - name: OSB_NAMESPACE
              value: crossplane-services
            - name: OSB_USERNAME
              value: test
            - name: OSB_PASSWORD
              value: changeMeEventually
            - name: OSB_SERVICE_IDS
              value: redis-k8s,3a385e26-cdfc-46bc-961b-69892684af8b,16379f5f-0c5f-4c55-a119-fd063af62919
            - # Used for Bearer Token Validation
              name: OSB_JWT_KEYS_JWK_URL
              value: https://auth.corp.internal/jwks
            - # Only configure this if plan upgrades are spcifically supported
              name: OSB_PLAN_UPDATE_SIZE_RULES
              value: "xsmall>small|xsmall>medium|small>medium"
          resources:
            limits:
              cpu: 500m
              memory: 128Mi
            requests:
              cpu: 200m
              memory: 64Mi
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
              scheme: HTTP
          readinessProbe:
            httpGet:
              path: /healthz
              port: http
              scheme: HTTP
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
          securityContext:
            runAsNonRoot: true
            readOnlyRootFilesystem: true
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      serviceAccountName: service-broker
      serviceAccount: service-broker
      schedulerName: default-scheduler
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  minReadySeconds: 30
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
---
kind: Service (5)
apiVersion: v1
metadata:
  name: service-broker-test
  namespace: service-broker
  labels:
    app.kubernetes.io/instance: test
    app.kubernetes.io/name: service-broker
    name: service-broker-test
spec:
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: http
  selector:
    app.kubernetes.io/instance: test
    app.kubernetes.io/name: service-broker
  type: ClusterIP
  sessionAffinity: None

1	This part of the YAML ensures that a Kubernetes namespace called `service-broker` exists.
2	This part of the YAML initiates the actual Deployment of the service broker.
3	This line defines the name of your Crossplane Service Broker. Don’t change it for now, as this guide relies on it being called `service-broker-test`.
4	See below for an explanation of the environment variables that are defined here.
5	In order for the service catalog to discover and access the Crossplane Service Broker, a Kubernetes `Service` is created. It conveniently also takes care of the load balancing between the two instances of the Crossplane Service Broker that get deployed.

Take note of the environment variables that are configured in the above deployment.yaml:

Variable Name Description Example Value

Variable Name	Description	Example Value
`OSB_SERVICE_IDS`	The Crossplane Service Broker must know which services it’s responsible for. The ID can be any arbitrary string, though often this is a UUID.	`redis-k8s,3a385e26-cdfc-46bc-961b-69892684af8b,16379f5f-0c5f-4c55-a119-fd063af62919`
`OSB_NAMESPACE`	This is the namespace in which the Crossplane Service Broker will create it the relevant Crossplane custom resources.	`crossplane-services`
`OSB_USERNAME`	This is the username which is used when doing Basic auth between the Service Catalog and the Service Broker. If you don’t use basic auth, choose a random string here.	`test`
`OSB_PASSWORD`	This is the password which is used when doing Basic auth between the Service Catalog and the Service Broker. If you don’t use basic auth, choose a random string here.	`changeMeEventually`
`OSB_JWT_KEYS_JWK_URL`	This URL is queried during the startup of the service broker. It contains the public keys in JWK-format that should be used to verify the validity of the JWT tokens. Learn more about this in the HTTP Bearer Token authentication how-to.	`https://auth.corp.internal/jwks`
`OSB_PLAN_UPDATE_SIZE_RULES`	Configures which plan size changes are valid. Takes a white-list in the form of `xsmall>small\|xsmall>medium`, which would allow upgrades from `xsmall` to `small` or `medium` and deny all other plan size changes. This requires explicit plan upgrade support from the Crossplane setup. Enabeling Plan Upgrades explains this setup in more detail. Don’t set this environment variable if this setup hasn’t been done for all services.	`xsmall>small\|xsmall>medium`
`OSB_PLAN_UPDATE_SLA_RULES`	Configures which SLA changes are valid. Takes a white-list in the form of `standard>premium\|premium>premium`, which would allow upgrades from `standard` to `premium` and back. This is the default configuration if this environment variable isn’t set.	`premium>standard\|standard>premium`

OSB_SERVICE_IDS

The Crossplane Service Broker must know which services it’s responsible for. The ID can be any arbitrary string, though often this is a UUID.

redis-k8s,3a385e26-cdfc-46bc-961b-69892684af8b,16379f5f-0c5f-4c55-a119-fd063af62919

OSB_NAMESPACE

This is the namespace in which the Crossplane Service Broker will create it the relevant Crossplane custom resources.

crossplane-services

OSB_USERNAME

This is the username which is used when doing Basic auth between the Service Catalog and the Service Broker. If you don’t use basic auth, choose a random string here.

test

OSB_PASSWORD

This is the password which is used when doing Basic auth between the Service Catalog and the Service Broker. If you don’t use basic auth, choose a random string here.

changeMeEventually

OSB_JWT_KEYS_JWK_URL

This URL is queried during the startup of the service broker. It contains the public keys in JWK-format that should be used to verify the validity of the JWT tokens.

Learn more about this in the HTTP Bearer Token authentication how-to.

https://auth.corp.internal/jwks

OSB_PLAN_UPDATE_SIZE_RULES

Configures which plan size changes are valid. Takes a white-list in the form of xsmall>small|xsmall>medium, which would allow upgrades from xsmall to small or medium and deny all other plan size changes.

This requires explicit plan upgrade support from the Crossplane setup. Enabeling Plan Upgrades explains this setup in more detail. Don’t set this environment variable if this setup hasn’t been done for all services.

xsmall>small|xsmall>medium

OSB_PLAN_UPDATE_SLA_RULES

Configures which SLA changes are valid. Takes a white-list in the form of standard>premium|premium>premium, which would allow upgrades from standard to premium and back. This is the default configuration if this environment variable isn’t set.

premium>standard|standard>premium

The Crossplane Service Broker only allows changes to a plans SLA or its size. Changing both the SLA and plan size at the same time isn’t allowed.

Both the SLA and plan size change rules are specified in a custom white-list format.

rule = rule|rule
rule = plan>plan

examples:
  planA>planB|planB>planC
  foo>bar
  planA>planB|plabB>planA|planC>planA

The list isn’t transitive. That means the rule A>B|B>C won’t allow the update from A to C in one step.

Get a Bearer Token

Below are curl commands shown. Depending on your configuration, your authentication server can only be reached from your Kubernetes cluster, but not from your personal computer. In these cases you can refer to the commands just below.

To run curl from your cluster, create an ad-hoc container using kubectl run:

kubectl run \
  --namespace default \
  -i --tty --rm \
  "curl-$(date +%s)" \
  --command /bin/sh \
  --image=quay.io/curlimages/curl --

Now you get a terminal in which you can then run curl commands, for example curl --version.

If you are more comfortable using wget, here’s a solution for that:

kubectl run \
  --namespace default \
  -i --tty --rm \
  "busybox-$(date +%s)" \
  --image=quay.io/prometheus/busybox --

Now you get a terminal in which you can then run wget commands, for example wget --version.

If you (or your company) use an OpenID compliant authentication server, you should be able to learn about the JWKS URL from the .well-known/openid-configuration JSON:

curl https://auth.corp.internal/.well-known/openid-configuration

The JSON will look like this:

{
  "issuer": "auth.corp.internal",
  "token_endpoint": "auth.corp.internal/token", (2)
  "jwks_uri": "auth.corp.internal/jwks", (1)
  "revocation_endpoint": "auth.corp.internal/revoke",
  "scopes_supported": ["openid"],
  "response_types_supported": [],
  "response_modes_supported": [],
  "grant_types_supported": ["client_credentials"],
  "acr_values_supported": [],
  "subject_types_supported": ["public"],
  "id_token_signing_alg_values_supported": ["ES256"],
  "token_endpoint_auth_methods_supported": ["client_secret_post"],
  "token_endpoint_auth_signing_alg_values_supported": ["ES256"],
  "claims_supported": [
    "aud",
    "exp",
    "iat",
    "iss",
    "allow_list",
    "cluster_id",
    "client_id",
    "sub"
  ],
  "code_challenge_methods_supported": []
}

1	Look for the `jwks_uri`. It’s the URL we’re interested in.
2	Also note down the `token_endpoint`. We’ll need it later.

JWK Store

To verify that the Crossplane Service Broker will be able to access this URL, check the response now:

curl https://auth.corp.internal/jwks

This should return a JSON like the following:

The values for x and y have been shortened in the example below.

{
  "keys": [
    {
      "kty": "EC",
      "crv": "P-256",
      "x": "6ze…",
      "y": "O5K…"
    }
  ]
}

Get a Bearer Token

In order to get a Bearer Token, you need to be in the possession of a client_id and a client_secret. Create these on your authentication server or request them from authorized personnel. Those should allow you to request a token from the authentication server on the /token endpoint:

The /token endpoint might be called differently on your server. See the section [_inspect_the_openid_configuration] above and look for the token_endpoint URL.

curl \
  --silent --request POST \
  --data "grant_type=client_credentials" \
  --data "client_id=950aaaa5-a656-4a8c-8515-aa505a550a52" \
  --data "client_secret=5a2924a5-050a-445a-aa5a-0a50a445a845" \
  "https://auth.corp.internal/token"

This usually returns a JSON like this:

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.zJrV44Lhr1Ck4vg1dMnldql0adLgut241jo0FbFXMlI", (1)
  "token_type": "Bearer",
  "expires_in": 600
}

1	The value-part here is your actual Bearer Token. The quotes (`"`) are just JSON syntax and don’t belong to the token itself.

You can check the content of the token on jwt.io.

Here’s a shortcut to grab the token directly:

kubectl run \
  --namespace default \
  --attach true --rm --quiet \
  "curl-$(date +%s)" \
  --image=docker.io/curlimages/curl -- \
  --silent --request POST \
  --data "grant_type=client_credentials" \
  --data "client_id=950aaaa5-a656-4a8c-8515-aa505a550a52" \
  --data "client_secret=5a2924a5-050a-445a-aa5a-0a50a445a845" \
  "https://auth.corp.internal/token" \
| jq -r .access_token

On macOS, you can also copy the token directly to your clipboard like this:

kubectl run \
  --namespace default \
  --attach true --rm --quiet \
  "curl-$(date +%s)" \
  --image=docker.io/curlimages/curl -- \
  --silent --request POST \
  --data "grant_type=client_credentials" \
  --data "client_id=950aaaa5-a656-4a8c-8515-aa505a550a52" \
  --data "client_secret=5a2924a5-050a-445a-aa5a-0a50a445a845" \
  "https://auth.corp.internal/token" \
| jq -r .access-token \
| pbcopy

Connect the Service Catalog to the Crossplane Service Broker

For the following instructions, change to the catalog namespace of your cluster:

kubectl config set-context --current --namespace=catalog

In order to connect the Service Catalog to the Crossplane Service Broker, we’ll have to provide the Service Catalog with some credentials. These credentials are used by the Service Catalog to identify itself to the Crossplane Service Broker. In this how-to, we focus on Bearer Token authentication. The advantage of the Bearer Token authentication is that the Crossplane Service Broker can restrict the offered services based on information provided in the Bearer Token.

The Setup HTTP Basic authentication how-to covers that.

The Service Catalog will read the Bearer Token from a Kubernetes secret resource. Therefore we need to get the Bearer Token from the authentication server and create a Kubernetes secret with it:

TOKEN=$(kubectl run \
  --namespace default \
  --attach --rm --quiet \
  "curl-$(date +%s)" \
  --image=docker.io/curlimages/curl -- \
  --silent --request POST \
  --data "grant_type=client_credentials" \
  --data "client_id=950aaaa5-a656-4a8c-8515-aa505a550a52" \
  --data "client_secret=5a2924a5-050a-445a-aa5a-0a50a445a845" \
  "https://auth.corp.internal/token" | jq -r .access_token)
printf "----BEGIN TOKEN----\n${TOKEN}\n----END TOKEN----\n\n"
kubectl create secret generic bearer-creds "--from-literal=token=${TOKEN}" --dry-run=client -o yaml | kubectl apply -f -

Don’t forget to change the values client_id, client_secret and the URL in the command above.

The token that’s issued by the authentication server may only be valid for a short time. (The token which was shown before in the HTTP Bearer Token authentication section was only valid for 600 seconds, that’s just 10 minutes.)

In that case you will need to deploy a kube-token-refresher. The Setup Kube Token Refresher how-to covers the setup. The token refresher will ensure that the service catalog always has a valid Bearer Token.

Register the Crossplane Service Broker at the Service Catalog

Now, finally, we must register the Crossplane Service Broker at the Service Catalog. The Service Catalog will immediately query the Crossplane Service Broker about the services it offers. It (the Service Catalog) will then updates its catalog of all the services that can be provided by our (and all other) registered Service Brokers.

To register the Crossplane Service Broker run the following command.

svcat register servicebroker-test --bearer-secret bearer-creds --url "http://service-broker-test.service-broker"

It should then be possible to interact with this instance of the Crossplane Service Broker through the Service Catalog:

# List all registered service brokers and their status
svcat get brokers

# Show the services that are available to order.
svcat marketplace

If you don’t need to use Bearer Token authentication, you may want to fall back to Basic authentication. This would typically be the case when your Crossplane Service Broker is only serving one team, group or customer anyways, or in other words: It’s a single tenant system instance. (The advantage of the Bearer Token authentication is that the Crossplane Service Broker can restrict the offered services based on information provided in the Bearer Token.)

See Setup HTTP Basic authentication on how to achieve that.