Enable LoadBalancer
services on cloudscale
Steps to enable
LoadBalancer
services on an OpenShift 4 cluster on cloudscale.
Starting situation
-
You already have an OpenShift 4 cluster on cloudscale
-
You have admin-level access to the cluster
-
The cluster is using Cilium as the network plugin
-
You want to enable LoadBalancer services
Prerequisites
-
ssh
-
curl
-
kubectl
-
yq
yq YAML processor (version 4 or higher) -
whois
Prepare IP range
-
Get a suitable public IP range. You can find all IP ranges owned by VSHN
whois -i mnt-by MNT-VSHN -T inetnum -r
Please coordinate with VSHN Corporate IT before using an IP range.
-
Update RIPE DB to reflect the new usage of the IP range. Add the IP range with a descriptive name.
-
Create a ticket with cloudscale to make the range available as a Floating Network in the project of the cluster.
Configure Loadbalancers
Add the chosen IP range as a loadbalancer address pool in the APPUiO hieradata.
Assuming you want to add the IP range 203.0.113.128/25
to the cluster c-cold-bush-7254
:
export CLUSTER_ID="c-cold-bush-7254"
export IP_RANGE="203.0.113.128/25"
git checkout -b feat/lbaas/loadbalancer-ip/${CLUSTER_ID}
mkdir -p lbaas/${CLUSTER_ID}
touch lbaas/${CLUSTER_ID}/lb.yaml
yq eval -i ".\"profile_openshift4_gateway::loadbalancer_ip_address_pools\" += [\"${IP_RANGE}\"]" lbaas/${CLUSTER_ID}/lb.yaml
git add lbaas/
git commit -a -m "Add loadbalancer IP range ${IP_RANGE} to ${CLUSTER_ID}"
git push --set-upstream origin feat/lbaas/loadbalancer-ip/${CLUSTER_ID}
Finally review and merge the MR in the APPUiO hieradata.
Setup BGP speaker
The following steps need to be done manually on both loadbalancer VMs
-
Install FRR
apt install frr
-
Enable bgp by setting
bgpd=yes
in/etc/frr/daemons
... bgpd=yes ospfd=no ospf6d=no ripd=no ripngd=no ...
-
Restart FRR
systemctl restart frr
-
Configure FRR
Enter the FRR shell with
vtysh
Configure BGP neighbors
conf t router bgp 64512 neighbor <ip-infra-node-0> remote-as 64512 (1) neighbor <ip-infra-node-1> remote-as 64512 neighbor <ip-infra-node-2> remote-as 64512 end write (2)
1 You can add any of the cluster node IPs. We propose to peer with each of the infrastructure nodes. 2 You need to write the configuration to persist
Configure Cilium
See the component-cilium documentation for detailed documentation of the bgp component parameters.
|
We configure Cilium via component-cilium
in the cluster’s tenant repo.
-
Ensure that
component-cilium
is at least v2.0.0 for the clusterparameters: components: cilium: version: v2.0.0
-
Enable Cilium’s BGP control plane feature in the cluster’s config
parameters: cilium: cilium_helm_values: LBIPAM: requireLBClass: true (1) bgp: enabled: true
1 Configure Cilium to only allocate LB IPs for services which have an io.cilium
load balancer class (spec.loadBalancerClass
). This ensures that Cilium BGP LB services can co-exist with LB services provisioned through the cloudscale cloud-controller-manager. -
Configure the Cilium BGP control plane
parameters: cilium: bgp: cluster_configs: lb-services: nodeSelector: (1) matchLabels: node-role.kubernetes.io/infra: "" bgpInstances: lbs: localASN: 64512 (2) peers: (3) lb-XY: peerAddress: 172.18.200.2 peerASN: 64512 peerConfigRef: name: lb-services (4) lb-ZW: peerAddress: 172.18.200.3 peerASN: 64512 peerConfigRef: name: lb-services (4) peer_configs: lb-services: (4) spec: gracefulRestart: (5) enabled: true restartTimeSeconds: 30 families: (6) unicast-v4: afi: ipv4 safi: unicast advertisements: matchLabels: cilium.syn.tools/advertise: bgp (7) advertisements: lb-services: metadata: labels: cilium.syn.tools/advertise: bgp (7) advertisements: lb-ips: (8) advertisementType: Service service: addresses: - LoadBalancerIP selector: matchLabels: appuio.io/load-balancer-class: cilium (9)
1 nodeSelector
defines on which nodes Cilium will start BGP speakers. Since we’ve setup the infra nodes as peers on the LBs, we select the infra nodes here.2 the local ASN for the Cilium speakers. This needs to match the ASN we configured for the infra nodes in frr
on the LBs.3 The peers for the Cilium BGP speakers. We configure both LBs here with their static IPs. If necessary, adjust the IPs for the cluster’s private network range. The ASN needs to match the FRR config on the LBs. We recommend using the LB host names as keys for the peers
configuration.4 Each peer references a CiliumBGPPeerConfig
. The name here must match an entry inbgp.peer_configs
.5 We explicitly configure graceful restart because Cilium doesn’t enable it by default. Graceful restart is required to ensure that FRR doesn’t withdraw the advertised routes immediately when the Cilium agent pod restarts. 6 We need to configure at least one address family to advertise. For LB services we want to advertise IPv4 unicast routes. 7 The family config has a label selector which must match the labels of the entry in bgp.advertisements
.8 We configure one advertisement for K8s service load balancer IPs. 9 This label selector defines for which services Cilium will announce ingress IPs. This can be changed to a NotIn
match expression on a non-existent service label to announce all services. We recommend to configure a matching label selector for the LB service IP pool to reduce the potential for partially managed LB services. -
Configure a Cilium LB IP pool
parameters: cilium: bgp: loadbalancer_ip_pools: lb-services: blocks: lb_ips: cidr: 203.0.113.128/25 (1) serviceSelector: matchLabels: appuio.io/load-balancer-class: cilium (2)
1 The public IP range you allocated for the cluster. 2 Service label selector which matches the one in the BGP advertisement configuration. -
Commit the changes and compile the cluster catalog
Test LoadBalancer
service
-
Apply a
LoadBalancer
service and a deployment:apiVersion: v1 kind: Service metadata: name: test-lb labels: appuio.io/load-balancer-class: cilium (1) spec: type: LoadBalancer loadBalancerClass: io.cilium/bgp-control-plane (2) ports: - port: 80 targetPort: 8080 protocol: TCP name: http selector: svc: test-lb --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: selector: matchLabels: svc: test-lb template: metadata: labels: svc: test-lb spec: containers: - name: web image: vshn/nginx imagePullPolicy: IfNotPresent ports: - containerPort: 8080 readinessProbe: httpGet: path: / port: 8080
1 We need to add label appuio.io/load-balancer-class=cliium
to the service, since we configure the Cilium LB IP pool and BGP advertisement to only process services with that label.2 We need to set spec.loadBalancerClass=io.cilium/bgp-control-plane
to tell Cilium to manage this service and to tell the cloudscale CCM to not manage this service. -
Observe that Cilium allocates an external IP for
test-lb
kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE test-lb LoadBalancer 172.20.0.5 203.0.113.132 80:30724/TCP 10s
-
Access the external IP
curl 203.0.113.132
Check the |
Troubleshoot
Check BGP peering
You can check if the BGP peering was successful by connecting to the loadbalancer VMs.
-
Enter the FRR shell with
vtysh
-
Show BGP summary.
show bgp summary
This should show you something similar to
BGP router identifier XXXX, local AS number 64512 vrf-id 0 BGP table version 6 RIB entries 5, using 920 bytes of memory Peers 3, using 61 KiB of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 172.18.200.137 4 64512 11120 11117 0 0 0 3d20h37m 3 172.18.200.157 4 64512 11120 11117 0 0 0 3d20h37m 3 172.18.200.218 4 64512 11119 11116 0 0 0 3d20h37m 3 Total number of neighbors 3
-
Show available routes
show ip route
This should include routes for the created LoadBalancer service.
If these checks look correct, the BGP setup works as expected. If you still can’t connect to the service, re-check the network policies and check if the Floating Network is assigned correctly.
If the neighbors or routes don’t show up correctly, follow the other troubleshooting steps.
Check BGP announcements
Next, check if Cilium sends out BGP announcements and whether they arrive at the loadbalancer VMs.
-
Check if Cilium sends out BGP announcements. In one of the Cilium DaemonSet pods run
tcpdump -n -i any tcp port 179
If Cilium sends out announcements to the correct IPs, it’s most likely setup correctly. If it doesn’t, there is an issue with Cilium. One thing to consider is that Cilium doesn’t automatically pick up updates of the
bgp-config
ConfigMap. Make sure to restart the Cilium DaemonSet pods if you change the configuration. -
Check if any BGP announcements arrive and are accepted. On one of the loadbalancer VMs run
tcpdump -n -i any tcp port 179
There should be packets coming in from the cluster nodes and they should be answered.
-
If no packets come in, check the connection between the cluster nodes and the loadbalancer VM.
-
If packets come in but aren’t answered, the issue might be the firewall setup. Check if the BGP port is open with
iptables -L
-
If the firewall accepts BGP announcements, check the FRR configuration. In the FRR shell run
show run
It should show the current running configuration which should look similar to
! frr version 7.2.1 frr defaults traditional hostname lb-1c log syslog informational no ipv6 forwarding service integrated-vtysh-config ! router bgp 64512 neighbor 172.18.200.137 remote-as 64512 neighbor 172.18.200.157 remote-as 64512 neighbor 172.18.200.218 remote-as 64512 ! line vty ! end
-