Enable LoadBalancer services on cloudscale.ch

Steps to enable LoadBalancer services on an OpenShift 4 cluster on cloudscale.ch.

Starting situation

  • You already have an OpenShift 4 cluster on cloudscale.ch

  • You have admin-level access to the cluster

  • The cluster is using Cilium as the network plugin

  • You want to enable LoadBalancer services


Prepare IP range

  1. Get a suitable public IP range. You can find all IP ranges owned by VSHN

    whois -i mnt-by MNT-VSHN -T inetnum -r

    Please coordinate with VSHN Corporate IT before using an IP range.

  2. Update RIPE DB to reflect the new usage of the IP range. Add the IP range with a descriptive name.

  3. Create a ticket with cloudscale.ch to make the range available as a Floating Network in the project of the cluster.

Configure Loadbalancers

Add the chosen IP range as a loadbalancer address pool in the APPUiO hieradata. Assuming you want to add the IP range to the cluster c-cold-bush-7254:

export CLUSTER_ID="c-cold-bush-7254"
export IP_RANGE=""

git checkout -b feat/lbaas/loadbalancer-ip/${CLUSTER_ID}

mkdir -p lbaas/${CLUSTER_ID}
touch lbaas/${CLUSTER_ID}/lb.yaml
yq eval -i ".\"profile_openshift4_gateway::loadbalancer_ip_address_pools\" += [\"${IP_RANGE}\"]" lbaas/${CLUSTER_ID}/lb.yaml

git add lbaas/
git commit -a -m "Add loadbalancer IP range ${IP_RANGE} to ${CLUSTER_ID}"
git push --set-upstream origin feat/lbaas/loadbalancer-ip/${CLUSTER_ID}

Finally review and merge the MR in the APPUiO hieradata.

Setup BGP speaker

The following steps need to be done manually on both loadbalancer VMs

  1. Install FRR

    apt install frr
  2. Enable bgp by setting bgpd=yes in /etc/frr/daemons

  3. Restart FRR

    systemctl restart frr
  4. Configure FRR

    Enter the FRR shell with


    Configure BGP neighbors

    conf t
    router bgp 64512
      neighbor <ip-infra-node-0> remote-as 64512 (1)
      neighbor <ip-infra-node-1> remote-as 64512
      neighbor <ip-infra-node-2> remote-as 64512
    write (2)
    1 You can add any of the cluster node IPs. We propose to peer with each of the infrastructure nodes.
    2 You need to write the configuration to persist

Configure Cilium

  1. Add a ConfigMap called bgp-config in the namespace cilium. This ConfigMap contains MetalLB configuration.

    Here’s an example ConfigMap.

    apiVersion: v1
    kind: ConfigMap
      name: bgp-config
      namespace: cilium
      config.yaml: |
          - peer-address: (1)
            peer-asn: 64512
            my-asn: 64512
          - peer-address: (1)
            peer-asn: 64512
            my-asn: 64512
          - name: default
            protocol: bgp
              - (2)
    1 IP addresses of the loadbalancers. If necessary, adjust these to match your cluster’s private network CIDR.
    2 The public IP range you allocated for the cluster
  2. Update the Cilium configuration to enable BGP. Our Cilium setup is configured through the CiliumConfig CRD called cilium in namespace cilium. The CRD contains Helm values. Update the CRD spec with

    apiVersion: cilium.io/v1alpha1
    kind: CiliumConfig
      name: cilium
      namespace: cilium
          loadbalancerIP: true
        enabled: true

Test LoadBalancer service

  1. Apply a LoadBalancer service and a deployment:

    apiVersion: v1
    kind: Service
      name: test-lb
      type: LoadBalancer
      - port: 80
        targetPort: 8080
        protocol: TCP
        name: http
        svc: test-lb
    apiVersion: apps/v1
    kind: Deployment
      name: nginx
          svc: test-lb
            svc: test-lb
          - name: web
            image: vshn/nginx
            imagePullPolicy: IfNotPresent
            - containerPort: 8080
                path: /
                port: 8080
  2. Observe that the Operator allocates an external IP for test-lb

    kubectl get svc
    NAME        TYPE          CLUSTER-IP  EXTERNAL-IP   PORT(S)       AGE
    test-lb     LoadBalancer 80:30724/TCP  10s
  3. Access the external IP


Check the NetworkPolicy in the target namespace. If the namespace doesn’t allow access from external nodes, everything will appear to work, but you won’t be able access the service from outside the cluster.


Check BGP peering

You can check if the BGP peering was successful by connecting to the loadbalancer VMs.

  1. Enter the FRR shell with

  2. Show BGP summary.

    show bgp summary

    This should show you something similar to

    BGP router identifier XXXX, local AS number 64512 vrf-id 0
    BGP table version 6
    RIB entries 5, using 920 bytes of memory
    Peers 3, using 61 KiB of memory
    Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd  4      64512   11120   11117        0    0    0 3d20h37m            3  4      64512   11120   11117        0    0    0 3d20h37m            3  4      64512   11119   11116        0    0    0 3d20h37m            3
    Total number of neighbors 3
  3. Show available routes

    show ip route

    This should include routes for the created LoadBalancer service.

If these checks look correct, the BGP setup works as expected. If you still can’t connect to the service, re-check the network policies and check if the Floating Network is assigned correctly.

If the neighbors or routes don’t show up correctly, follow the other troubleshooting steps.

Check BGP announcements

Next, check if Cilium sends out BGP announcements and whether they arrive at the loadbalancer VMs.

  1. Check if Cilium sends out BGP announcements. In one of the Cilium DaemonSet pods run

    tcpdump -n -i any tcp port 179

    If Cilium sends out announcements to the correct IPs, it’s most likely setup correctly. If it doesn’t, there is an issue with Cilium. One thing to consider is that Cilium doesn’t automatically pick up updates of the bgp-config ConfigMap. Make sure to restart the Cilium DaemonSet pods if you change the configuration.

  2. Check if any BGP announcements arrive and are accepted. On one of the loadbalancer VMs run

    tcpdump -n -i any tcp port 179

    There should be packets coming in from the cluster nodes and they should be answered.

    1. If no packets come in, check the connection between the cluster nodes and the loadbalancer VM.

    2. If packets come in but aren’t answered, the issue might be the firewall setup. Check if the BGP port is open with

      iptables -L
    3. If the firewall accepts BGP announcements, check the FRR configuration. In the FRR shell run

      show run

      It should show the current running configuration which should look similar to

      frr version 7.2.1
      frr defaults traditional
      hostname lb-1c
      log syslog informational
      no ipv6 forwarding
      service integrated-vtysh-config
      router bgp 64512
       neighbor remote-as 64512
       neighbor remote-as 64512
       neighbor remote-as 64512
      line vty