Access nodes for debugging

Healthy nodes

You can access nodes for debugging with oc debug node as long as the Kubernetes API is available and pods can be scheduled on the node. If this isn’t the case see the Unhealthy nodes.

In order to ensure that the debug pod is scheduled correctly, use the following command:

nodename=<NODE> (1)
oc --as=cluster-admin -n syn-debug-nodes debug "node/${nodename}" (2)
1 The name of the node to debug. Use the names shown in oc get nodes.
2 Impersonating cluster-admin with --as=cluster-admin ensures that you have sufficient permissions to schedule the debug pod. Selecting namespace syn-debug-nodes with -n syn-debug-nodes ensures that the debug pod can be scheduled on any node in the cluster.

Unhealthy nodes

To access nodes when the Kubernetes API isn’t available, or to investigate a node on which new pods can’t be started, you can directly access the node over SSH. Follow the steps below to access a node over SSH.

  1. Extract the node SSH key from Vault and add it to your SSH agent

    Check the cluster’s documentation to find the node SSH key for infrastructures other than and Exoscale.
    export CLOUD=<cloud-provider-id> (1)
    export CLUSTER_ID=<cluster-id>
    export TENANT_ID=<tenant-id>
    export VAULT_ADDR=
    vault login -method=oidc
    vault kv get -format json clusters/kv/${TENANT_ID}/${CLUSTER_ID}/${CLOUD}/ssh | \
      jq -r '' | base64 -d > ${CLUSTER_ID}_ssh
    chmod 600 ${CLUSTER_ID}_ssh
    ssh-add ${CLUSTER_ID}_ssh
    1 Use cloudscale for, and exoscale for Exoscale.
  2. Use one of the Puppet-managed LBs as a SSH jumphost.

    Strictly speaking, this isn’t necessary for Exoscale, as the nodes are directly reachable.
    For clusters on other infrastructures, check the cluster’s network documentation to find a host from which you can access the node network.
    export (1)
    <1> Replace with the FQDN of one of the Puppet-managed LBs
  3. SSH into the node

    NODE_IP=<node-ip> (1)
    ssh -J ${JUMPHOST} core@${NODE_IP}
    1 Replace with the node’s IP address.

    If the Kubernetes API is reachable, you can extract the node IP with

    kubectl get node <NODENAME> -o json | \
      jq -r '.status.addresses[] | select(.type == "InternalIP") | .address'

    User core has full sudo permissions. You can become root with sudo -i.

  4. After you’re done, remove the node SSH key from the SSH agent

    ssh-add -d ${CLUSTER_ID}_ssh
    rm ${CLUSTER_ID}_ssh