Migrate to Cilium CNI


  • cluster-admin privileges

  • kubectl

  • jq

  • curl

  • Working commodore command

Prepare for migration

Make sure that your $KUBECONFIG points to the cluster you want to migrate before starting.
  1. Create alertmanager silence

        kubectl --as=cluster-admin -n openshift-monitoring exec \
        sts/alertmanager-main -- amtool --alertmanager.url=http://localhost:9093 \
        silence add alertname!=Watchdog --duration="3h" -c "cilium migration"
    echo $silence_id
  2. Select cluster

    export CLUSTER_ID=c-cluster-id-1234 (1)
    export COMMODORE_API_URL=https://api.syn.vshn.net (2)
    export TENANT_ID=$(curl -sH "Authorization: Bearer $(commodore fetch-token)" \
      "${COMMODORE_API_URL}/clusters/${CLUSTER_ID}" | jq -r '.tenant')
    1 Replace with the Project Syn cluster ID of the cluster to migrate
    2 Replace with the Lieutenant API on which the cluster is registered
  3. Disable ArgoCD auto sync for components openshift4-nodes and openshift-upgrade-controller

    kubectl --as=cluster-admin -n syn patch apps root --type=json \
      -p '[{"op":"replace", "path":"/spec/syncPolicy", "value": {}}]'
    kubectl --as=cluster-admin -n syn patch apps openshift4-nodes --type=json \
      -p '[{"op":"replace", "path":"/spec/syncPolicy", "value": {}}]'
    kubectl --as=cluster-admin -n syn patch apps openshift-upgrade-controller --type=json \
      -p '[{"op":"replace", "path":"/spec/syncPolicy", "value": {}}]'
  4. Disable the cluster-network-operator. This is necessary to ensure that we can migrate to Cilium without the cluster-network-operator trying to interfere. We also need to scale down the upgrade controller, so that we can patch the ClusterVersion object.

    kubectl --as=cluster-admin -n appuio-openshift-upgrade-controller \
      scale deployment openshift-upgrade-controller-controller-manager --replicas=0
    kubectl --as=cluster-admin patch clusterversion version \
      --type=merge \
      -p '
          "kind": "Deployment",
          "group": "apps",
          "name": "network-operator",
          "namespace": "openshift-network-operator",
          "unmanaged": true
    kubectl --as=cluster-admin -n openshift-network-operator \
      scale deploy network-operator --replicas=0
  5. Verify that the network operator has been scaled down.

    kubectl -n openshift-network-operator get pods (1)
    1 This should return No resources found in openshift-network-operator namespace.

    If the operator is still running, check the following conditions:

    • The APPUiO OpenShift upgrade controller must be scaled down.

    • The ClusterVersion object must have an override to make the network operator deployment unmanaged.

  6. Remove network operator applied state

    kubectl --as=cluster-admin -n openshift-network-operator \
      delete configmap applied-cluster
  7. Pause all machine config pools

    for mcp in $(kubectl get mcp -o name); do
    kubectl --as=cluster-admin patch $mcp --type=merge -p '{"spec": {"paused": true}}'

Migrate to Cilium

  1. Get local cluster working directory

    commodore catalog compile "$CLUSTER_ID" (1)
    1 We recommend switching to an empty directory to run this command. Alternatively, switch to your existing directory for the cluster.
  2. Enable component cilium

    pushd inventory/classes/"${TENANT_ID}"
    yq -i '.applications += "cilium"' "${CLUSTER_ID}.yml"
  3. Update upstreamRules for monitoring

    yq -i ".parameters.openshift4_monitoring.upstreamRules.networkPlugin = \"cilium\"" \
  4. Update component networkpolicy config

    yq eval -i '.parameters.networkpolicy.networkPlugin = "cilium"' \
    yq eval -i '.parameters.networkpolicy.ignoredNamespaces = ["openshift-oauth-apiserver"]' \
  5. Verify that the cluster’s api-int DNS record exists

    export CLUSTER_DOMAIN=$(kubectl get dns cluster -ojsonpath='{.spec.baseDomain}')
    kubectl --as=cluster-admin -n openshift-dns exec ds/node-resolver -- nslookup api-int.${CLUSTER_DOMAIN}

    If the command doesn’t return a valid record for api-int, the Cilium config needs to be adjusted to use the api record to ensure that the Kubernetes API can be accessed.

    yq -i '.parameters.cilium.cilium_helm_values.k8sServiceHost="api.${openshift:baseDomain}"' \
  6. Configure component cilium.

    Configure the cluster Pod and Service CIDRs
    POD_CIDR=$(kubectl get network.config cluster \
      -o jsonpath='{.spec.clusterNetwork[0].cidr}')
    HOST_PREFIX=$(kubectl get network.config cluster \
      -o jsonpath='{.spec.clusterNetwork[0].hostPrefix}')
    yq -i '.parameters.cilium.cilium_helm_values.ipam.operator.clusterPoolIPv4MaskSize = "'"${HOST_PREFIX}"'"' \
    yq -i '.parameters.cilium.cilium_helm_values.ipam.operator.~clusterPoolIPv4PodCIDRList = "'"${POD_CIDR}"'"' \
  7. Commit changes

    git commit -am "Migrate ${CLUSTER_ID} to Cilium"
    git push origin master
  8. Compile catalog

    commodore catalog compile "${CLUSTER_ID}"
  9. Patch cluster network config

    Only execute this step after you’ve paused all machine config pools. Otherwise, nodes may reboot into a state where they’re stuck in NotReady.

    kubectl --as=cluster-admin patch network.config cluster \
      --type=merge -p '{"spec":{"networkType":"Cilium"},"status":null}'
    kubectl --as=cluster-admin patch network.operator cluster \
      --type=merge -p '{"spec":{"defaultNetwork":{"type":"Cilium"}},"status":null}'
  10. Apply Cilium manifests. We need to execute the apply twice, since the first apply will fail to create the CiliumConfig resource.

    kubectl --as=cluster-admin apply -Rf catalog/manifests/cilium/
    kubectl --as=cluster-admin apply -Rf catalog/manifests/cilium/
  11. Wait until Cilium CNI is up and running

    kubectl -n cilium get pods -w

Finalize migration

  1. Re-enable cluster network operator

    This will remove the previously active CNI plugin and will deploy the kube-proxy daemonset. As soon as you complete this step, existing pods may go into CrashLoopBackOff since they were started with CNI IPs managed by the old network plugin.

    kubectl --as=cluster-admin -n openshift-network-operator \
      scale deployment network-operator --replicas=1
    kubectl --as=cluster-admin patch clusterversion version \
     --type=merge -p '{"spec":{"overrides":null}}'
  2. Unpause MCPs

    for mcp in $(kubectl get mcp -o name); do
    kubectl --as=cluster-admin patch $mcp --type=merge -p '{"spec":{"paused":false}}'

    You may need to grab the cluster-admin credentials to complete this step since the OpenShift OAuth components may be unavailable until they’re restarted with Cilium-managed IPs.

    You may want to restart the multus daemonset once the old CNI pods are removed.

    kubectl --as=cluster-admin -n openshift-multus rollout restart ds/multus

    It may be necessary to force drain nodes manually to allow the machine-config-operator to reboot the nodes. Use kubectl --as=cluster-admin drain --ignore-daemonsets --delete-emptydir-data --force --disable-eviction to circumvent PDB violations if necessary.

    Start with a master node, and ensure that the machine-config-operator is running on that master node after it’s been drained and rebooted.

  3. Compile and push catalog

    commodore catalog compile "${CLUSTER_ID}" --push
  4. Re-enable ArgoCD auto sync

    kubectl --as=cluster-admin -n syn patch apps root --type=json \
      -p '[{
        "value": {"automated": {"prune": true, "selfHeal": true}}

Cleanup alert silence

  1. Expire alertmanager silence

    kubectl --as=cluster-admin -n openshift-monitoring exec sts/alertmanager-main --\
        amtool --alertmanager.url=http://localhost:9093 silence expire $silence_id