Enable LoadBalancer services on cloudscale

Steps to enable LoadBalancer services on an OpenShift 4 cluster on cloudscale.

Starting situation

  • You already have an OpenShift 4 cluster on cloudscale

  • You have admin-level access to the cluster

  • The cluster is using Cilium as the network plugin

  • You want to enable LoadBalancer services

Prerequisites

Prepare IP range

  1. Get a suitable public IP range. You can find all IP ranges owned by VSHN

    whois -i mnt-by MNT-VSHN -T inetnum -r

    Please coordinate with VSHN Corporate IT before using an IP range.

  2. Update RIPE DB to reflect the new usage of the IP range. Add the IP range with a descriptive name.

  3. Create a ticket with cloudscale to make the range available as a Floating Network in the project of the cluster.

Configure Loadbalancers

Add the chosen IP range as a loadbalancer address pool in the APPUiO hieradata. Assuming you want to add the IP range 203.0.113.128/25 to the cluster c-cold-bush-7254:

export CLUSTER_ID="c-cold-bush-7254"
export IP_RANGE="203.0.113.128/25"

git checkout -b feat/lbaas/loadbalancer-ip/${CLUSTER_ID}

mkdir -p lbaas/${CLUSTER_ID}
touch lbaas/${CLUSTER_ID}/lb.yaml
yq eval -i ".\"profile_openshift4_gateway::loadbalancer_ip_address_pools\" += [\"${IP_RANGE}\"]" lbaas/${CLUSTER_ID}/lb.yaml

git add lbaas/
git commit -a -m "Add loadbalancer IP range ${IP_RANGE} to ${CLUSTER_ID}"
git push --set-upstream origin feat/lbaas/loadbalancer-ip/${CLUSTER_ID}

Finally review and merge the MR in the APPUiO hieradata.

Setup BGP speaker

The following steps need to be done manually on both loadbalancer VMs

  1. Install FRR

    apt install frr
  2. Enable bgp by setting bgpd=yes in /etc/frr/daemons

    ...
    bgpd=yes
    ospfd=no
    ospf6d=no
    ripd=no
    ripngd=no
    ...
  3. Restart FRR

    systemctl restart frr
  4. Configure FRR

    Enter the FRR shell with

    vtysh

    Configure BGP neighbors

    conf t
    router bgp 64512
      neighbor <ip-infra-node-0> remote-as 64512 (1)
      neighbor <ip-infra-node-1> remote-as 64512
      neighbor <ip-infra-node-2> remote-as 64512
      end
    write (2)
    1 You can add any of the cluster node IPs. We propose to peer with each of the infrastructure nodes.
    2 You need to write the configuration to persist

Configure Cilium

See the component-cilium documentation for detailed documentation of the bgp component parameters.

We configure Cilium via component-cilium in the cluster’s tenant repo.

  1. Ensure that component-cilium is at least v2.0.0 for the cluster

    parameters:
      components:
        cilium:
          version: v2.0.0
  2. Enable Cilium’s BGP control plane feature in the cluster’s config

    parameters:
      cilium:
        cilium_helm_values:
          LBIPAM:
            requireLBClass: true (1)
        bgp:
          enabled: true
    1 Configure Cilium to only allocate LB IPs for services which have an io.cilium load balancer class (spec.loadBalancerClass). This ensures that Cilium BGP LB services can co-exist with LB services provisioned through the cloudscale cloud-controller-manager.
  3. Configure the Cilium BGP control plane

    parameters:
      cilium:
        bgp:
          cluster_configs:
            lb-services:
              nodeSelector: (1)
                matchLabels:
                  node-role.kubernetes.io/infra: ""
              bgpInstances:
                lbs:
                  localASN: 64512 (2)
                  peers: (3)
                    lb-XY:
                      peerAddress: 172.18.200.2
                      peerASN: 64512
                      peerConfigRef:
                        name: lb-services (4)
                    lb-ZW:
                      peerAddress: 172.18.200.3
                      peerASN: 64512
                      peerConfigRef:
                        name: lb-services (4)
          peer_configs:
            lb-services: (4)
              spec:
                gracefulRestart: (5)
                  enabled: true
                  restartTimeSeconds: 30
              families: (6)
                unicast-v4:
                  afi: ipv4
                  safi: unicast
                  advertisements:
                    matchLabels:
                      cilium.syn.tools/advertise: bgp (7)
          advertisements:
            lb-services:
              metadata:
                labels:
                  cilium.syn.tools/advertise: bgp (7)
              advertisements:
                lb-ips: (8)
                  advertisementType: Service
                  service:
                    addresses:
                      - LoadBalancerIP
                  selector:
                    matchLabels:
                      appuio.io/load-balancer-class: cilium (9)
    1 nodeSelector defines on which nodes Cilium will start BGP speakers. Since we’ve setup the infra nodes as peers on the LBs, we select the infra nodes here.
    2 the local ASN for the Cilium speakers. This needs to match the ASN we configured for the infra nodes in frr on the LBs.
    3 The peers for the Cilium BGP speakers. We configure both LBs here with their static IPs. If necessary, adjust the IPs for the cluster’s private network range. The ASN needs to match the FRR config on the LBs. We recommend using the LB host names as keys for the peers configuration.
    4 Each peer references a CiliumBGPPeerConfig. The name here must match an entry in bgp.peer_configs.
    5 We explicitly configure graceful restart because Cilium doesn’t enable it by default. Graceful restart is required to ensure that FRR doesn’t withdraw the advertised routes immediately when the Cilium agent pod restarts.
    6 We need to configure at least one address family to advertise. For LB services we want to advertise IPv4 unicast routes.
    7 The family config has a label selector which must match the labels of the entry in bgp.advertisements.
    8 We configure one advertisement for K8s service load balancer IPs.
    9 This label selector defines for which services Cilium will announce ingress IPs. This can be changed to a NotIn match expression on a non-existent service label to announce all services. We recommend to configure a matching label selector for the LB service IP pool to reduce the potential for partially managed LB services.
  4. Configure a Cilium LB IP pool

    parameters:
      cilium:
        bgp:
          loadbalancer_ip_pools:
            lb-services:
              blocks:
                lb_ips:
                  cidr: 203.0.113.128/25 (1)
              serviceSelector:
                matchLabels:
                  appuio.io/load-balancer-class: cilium (2)
    1 The public IP range you allocated for the cluster.
    2 Service label selector which matches the one in the BGP advertisement configuration.
  5. Commit the changes and compile the cluster catalog

Test LoadBalancer service

  1. Apply a LoadBalancer service and a deployment:

    apiVersion: v1
    kind: Service
    metadata:
      name: test-lb
      labels:
        appuio.io/load-balancer-class: cilium (1)
    spec:
      type: LoadBalancer
      loadBalancerClass: io.cilium/bgp-control-plane (2)
      ports:
      - port: 80
        targetPort: 8080
        protocol: TCP
        name: http
      selector:
        svc: test-lb
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx
    spec:
      selector:
        matchLabels:
          svc: test-lb
      template:
        metadata:
          labels:
            svc: test-lb
        spec:
          containers:
          - name: web
            image: vshn/nginx
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 8080
            readinessProbe:
              httpGet:
                path: /
                port: 8080
    1 We need to add label appuio.io/load-balancer-class=cliium to the service, since we configure the Cilium LB IP pool and BGP advertisement to only process services with that label.
    2 We need to set spec.loadBalancerClass=io.cilium/bgp-control-plane to tell Cilium to manage this service and to tell the cloudscale CCM to not manage this service.
  2. Observe that Cilium allocates an external IP for test-lb

    kubectl get svc
    
    NAME        TYPE          CLUSTER-IP  EXTERNAL-IP   PORT(S)       AGE
    test-lb     LoadBalancer  172.20.0.5  203.0.113.132 80:30724/TCP  10s
  3. Access the external IP

    curl 203.0.113.132

Check the NetworkPolicy in the target namespace. If the namespace doesn’t allow access from external nodes, everything will appear to work, but you won’t be able access the service from outside the cluster.

Troubleshoot

Check BGP peering

You can check if the BGP peering was successful by connecting to the loadbalancer VMs.

  1. Enter the FRR shell with

    vtysh
  2. Show BGP summary.

    show bgp summary

    This should show you something similar to

    BGP router identifier XXXX, local AS number 64512 vrf-id 0
    BGP table version 6
    RIB entries 5, using 920 bytes of memory
    Peers 3, using 61 KiB of memory
    
    Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd
    172.18.200.137  4      64512   11120   11117        0    0    0 3d20h37m            3
    172.18.200.157  4      64512   11120   11117        0    0    0 3d20h37m            3
    172.18.200.218  4      64512   11119   11116        0    0    0 3d20h37m            3
    
    Total number of neighbors 3
  3. Show available routes

    show ip route

    This should include routes for the created LoadBalancer service.

If these checks look correct, the BGP setup works as expected. If you still can’t connect to the service, re-check the network policies and check if the Floating Network is assigned correctly.

If the neighbors or routes don’t show up correctly, follow the other troubleshooting steps.

Check BGP announcements

Next, check if Cilium sends out BGP announcements and whether they arrive at the loadbalancer VMs.

  1. Check if Cilium sends out BGP announcements. In one of the Cilium DaemonSet pods run

    tcpdump -n -i any tcp port 179

    If Cilium sends out announcements to the correct IPs, it’s most likely setup correctly. If it doesn’t, there is an issue with Cilium. One thing to consider is that Cilium doesn’t automatically pick up updates of the bgp-config ConfigMap. Make sure to restart the Cilium DaemonSet pods if you change the configuration.

  2. Check if any BGP announcements arrive and are accepted. On one of the loadbalancer VMs run

    tcpdump -n -i any tcp port 179

    There should be packets coming in from the cluster nodes and they should be answered.

    1. If no packets come in, check the connection between the cluster nodes and the loadbalancer VM.

    2. If packets come in but aren’t answered, the issue might be the firewall setup. Check if the BGP port is open with

      iptables -L
    3. If the firewall accepts BGP announcements, check the FRR configuration. In the FRR shell run

      show run

      It should show the current running configuration which should look similar to

      !
      frr version 7.2.1
      frr defaults traditional
      hostname lb-1c
      log syslog informational
      no ipv6 forwarding
      service integrated-vtysh-config
      !
      router bgp 64512
       neighbor 172.18.200.137 remote-as 64512
       neighbor 172.18.200.157 remote-as 64512
       neighbor 172.18.200.218 remote-as 64512
      !
      line vty
      !
      end