Install OpenShift 4 on STACKIT
Steps to install an OpenShift 4 cluster on STACKIT.
These steps follow the Installing a cluster on bare metal docs to set up a user provisioned installation (UPI). Terraform is used to provision the cloud infrastructure.
The install instructions for STACKIT are in early alpha state, and are still being tested and improved. |
The commands are idempotent and can be retried if any of the steps fail. The certificates created during bootstrap are only valid for 24h. So make sure you complete these steps within 24h. |
This how-to guide is still a work in progress and will change. It’s currently very specific to VSHN and needs further changes to be more generic. |
Starting situation
-
You already have a Tenant and its git repository
-
You have a CCSP Red Hat login and are logged into Red Hat Openshift Cluster Manager
Don’t use your personal account to login to the cluster manager for installation. -
You want to register a new cluster in Lieutenant and are about to install Openshift 4 on STACKIT
Prerequisites
-
jq
-
yq
yq YAML processor (version 4 or higher - use the go version by mikefarah, not the jq wrapper by kislyuk) -
vault
Vault CLI -
curl
-
emergency-credentials-receive
Install instructions -
gzip
-
docker
-
mc
>=RELEASE.2021-07-27T06-46-19Z
Minio client (aliased tomc
if necessary) -
stackit
>=v0.30.0
STACKIT cli
Make sure the minor version of |
Cluster Installation
Register the new OpenShift 4 cluster in Lieutenant.
Set cluster facts
For customer clusters, set the following cluster facts in Lieutenant:
-
sales_order
: Name of the sales order to which the cluster is billed, such asS10000
-
service_level
: Name of the service level agreement for this cluster, such asguaranteed-availability
-
access_policy
: Access-Policy of the cluster, such asregular
orswissonly
-
release_channel
: Name of the syn component release channel to use, such asstable
-
maintenance_window
: Pick the appropriate upgrade schedule, such asmonday-1400
for test clusters,tuesday-1000
for prod orcustom
to not (yet) enable maintenance -
cilium_addons
: Comma-separated list of cilium addons the customer gets billed for, such asadvanced_networking
ortetragon
. Set toNONE
if no addons should be billed.
Set up Keycloak service
-
Create a Keycloak service
Use control.vshn.net/vshn/services/_create to create a service. The name and ID must be clusters name. For the optional URL use the OpenShift console URL.
Configure input
Ensure you have access to the appropriate STACKIT project and make note of its ID (can be found in the Resource Manager).
export STACKIT_PROJECT_ID="YOUR_PROJECT_ID" # STACKIT project UUID, found in the Resource Manager on portal.stackit.cloud
stackit auth login
# From https://git.vshn.net/-/user_settings/personal_access_tokens, "api" scope is sufficient
export GITLAB_TOKEN=<gitlab-api-token>
export GITLAB_USER=<gitlab-user-name>
# For example: https://api.syn.vshn.net
# IMPORTANT: do NOT add a trailing `/`. Commands below will fail.
export COMMODORE_API_URL=<lieutenant-api-endpoint>
# Set Project Syn cluster and tenant ID
export CLUSTER_ID=<lieutenant-cluster-id> # Looks like: c-<something>
export TENANT_ID=$(curl -sH "Authorization: Bearer $(commodore fetch-token)" ${COMMODORE_API_URL}/clusters/${CLUSTER_ID} | jq -r .tenant)
export BASE_DOMAIN=<your-base-domain> # customer-provided base domain without cluster name, e.g. "zrh.customer.vshnmanaged.net"
export PULL_SECRET='<redhat-pull-secret>' # As copied from https://cloud.redhat.com/openshift/install/pull-secret "Copy pull secret". value must be inside quotes.
For BASE_DOMAIN
explanation, see DNS Scheme.
Set up S3 bucket for cluster bootstrap
-
Enable STACKIT Object Storage
stackit object-storage enable --project-id "${STACKIT_PROJECT_ID}" -y
-
Create S3 bucket
export BUCKET="${CLUSTER_ID}-bootstrap" stackit object-storage bucket create "${BUCKET}" --project-id "${STACKIT_PROJECT_ID}" -y
-
Create S3 user
credentials_group="`stackit object-storage credentials-group list --project-id "${STACKIT_PROJECT_ID}" -o json | jq -r '.[] | select (.displayName == "default") | .credentialsGroupId'`" credential="`stackit object-storage credentials create --credentials-group-id "${credentials_group}" --project-id "${STACKIT_PROJECT_ID}" -y -o json`" export ACCESS_KEY="`echo "$credential" | jq -r .accessKeyId`" export SECRET_KEY="`echo "$credential" | jq -r .secretAccessKey`"
-
Configure the Minio client
export REGION=$(curl -sH "Authorization: Bearer $(commodore fetch-token)" ${COMMODORE_API_URL}/clusters/${CLUSTER_ID} | jq -r .facts.region) mc config host add "${CLUSTER_ID}" "https://object.storage.${REGION}.onstackit.cloud" "$ACCESS_KEY" "$SECRET_KEY" mc mb --ignore-existing \ "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition"
Upload Red Hat CoreOS image
-
Check if image already exists in the project
stackit curl https://iaas.api.eu01.stackit.cloud/v1/projects/${STACKIT_PROJECT_ID}/images/ | jq '.items[] | select(.name == "rhcos-4.17")'
If this command returns an image, make note of its Image ID and skip the next steps.
-
Fetch the latest Red Hat CoreOS image
curl -L https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.17/4.17.2/rhcos-4.17.2-x86_64-openstack.x86_64.qcow2.gz | gzip -d > rhcos-4.17.qcow2
-
Upload the image to STACKIT
response="`stackit curl -X POST -H "Content-Type: application/json" --data '{"active": true, "diskFormat": "qcow2","name": "rhcos-4.17"}' https://iaas.api.eu01.stackit.cloud/v1beta1/projects/${STACKIT_PROJECT_ID}/images`" export IMAGE_ID="`echo $response | jq -r .id`" IMAGE_UPLOAD_URL="`echo $response | jq -r .uploadUrl`" curl -X PUT -H 'Content-Type: binary/octet-stream' --upload-file rhcos-4.17.qcow2 "$IMAGE_UPLOAD_URL" | cat
-
Wait for image import to complete
echo "Waiting for image to become available..." while [[ `stackit curl https://iaas.api.eu01.stackit.cloud/v1/projects/"${STACKIT_PROJECT_ID}"/images/"${IMAGE_ID}" | jq -r .status` == "CREATING" ]] ; do echo -n . ; sleep 4 ; done echo -n "Status is now " ; stackit curl https://iaas.api.eu01.stackit.cloud/v1/projects/"${STACKIT_PROJECT_ID}"/images/"${IMAGE_ID}" | jq .status
Set secrets in Vault
export VAULT_ADDR=https://vault-prod.syn.vshn.net
vault login -method=oidc
# Store S3 credentials in Vault
vault kv put clusters/kv/${TENANT_ID}/${CLUSTER_ID}/stackit/storage_iam \
s3_access_key=${ACCESS_KEY} s3_secret_key=${SECRET_KEY}
# Generate an HTTP secret for the registry
vault kv put clusters/kv/${TENANT_ID}/${CLUSTER_ID}/registry \
httpSecret=$(LC_ALL=C tr -cd "A-Za-z0-9" </dev/urandom | head -c 128)
# Generate a master password for K8up backups
vault kv put clusters/kv/${TENANT_ID}/${CLUSTER_ID}/global-backup \
password=$(LC_ALL=C tr -cd "A-Za-z0-9" </dev/urandom | head -c 32)
# Generate a password for the cluster object backups
vault kv put clusters/kv/${TENANT_ID}/${CLUSTER_ID}/cluster-backup \
password=$(LC_ALL=C tr -cd "A-Za-z0-9" </dev/urandom | head -c 32)
Prepare Cluster Repository
Starting with this section, we recommend that you change into a clean directory (for example a directory in your home). |
Check Running Commodore for details on how to run commodore. |
-
Prepare Commodore inventory.
mkdir -p inventory/classes/ git clone $(curl -sH"Authorization: Bearer $(commodore fetch-token)" "${COMMODORE_API_URL}/tenants/${TENANT_ID}" | jq -r '.gitRepo.url') inventory/classes/${TENANT_ID}
-
Configure the cluster’s domain in Project Syn
export CLUSTER_DOMAIN="${CLUSTER_ID}.${BASE_DOMAIN}" (1)
1 Adjust this as necessary if you’re using a non-standard cluster domain. The cluster domain configured here must be correct. The value is used to configure how Cilium connects to the cluster’s K8s API.
pushd "inventory/classes/${TENANT_ID}/" yq eval -i ".parameters.openshift.baseDomain = \"${CLUSTER_DOMAIN}\"" \ ${CLUSTER_ID}.yml git commit -a -m "Configure cluster domain for ${CLUSTER_ID}"
-
Include
openshift4.yml
in the cluster’s config if it existsFor some tenants, this may already configure some of the settings shown in this how-to. if ls openshift4.y*ml 1>/dev/null 2>&1; then yq eval -i '.classes += ".openshift4"' ${CLUSTER_ID}.yml; git commit -a -m "Include openshift4 class for ${CLUSTER_ID}" fi
-
Add Cilium to cluster configuration
These instructions assume that Cilium is configured to use
api-int.${CLUSTER_DOMAIN}:6443
to connect to the cluster’s K8s API. To ensure that that’s the case, add the configuration shown below somewhere in the Project Syn config hierarchy.parameters: cilium: cilium_helm_values: k8sServiceHost: api-int.${openshift:baseDomain} k8sServicePort: "6443"
For VSHN, this configuration is set in the Commodore global defaults (internal).
If you have a non-standard pod network, you need to ensure to include this in the configuration.
parameters: cilium: cilium_helm_values: ipam: operator: ~clusterPoolIPv4PodCIDRList: - <POD_NETWORK_CIDR>
yq eval -i '.applications += ["cilium"]' ${CLUSTER_ID}.yml yq eval -i '.parameters.networkpolicy.networkPlugin = "cilium"' ${CLUSTER_ID}.yml yq eval -i '.parameters.networkpolicy.ignoredNamespaces = ["openshift-oauth-apiserver"]' ${CLUSTER_ID}.yml yq eval -i '.parameters.openshift4_monitoring.upstreamRules.networkPlugin = "cilium"' ${CLUSTER_ID}.yml yq eval -i '.parameters.openshift.infraID = "TO_BE_DEFINED"' ${CLUSTER_ID}.yml yq eval -i '.parameters.openshift.clusterID = "TO_BE_DEFINED"' ${CLUSTER_ID}.yml git commit -a -m "Add Cilium addon to ${CLUSTER_ID}" git push popd
-
Compile catalog
commodore catalog compile ${CLUSTER_ID} --push -i \ --dynamic-fact kubernetesVersion.major=$(echo "1.30" | awk -F. '{print $1}') \ --dynamic-fact kubernetesVersion.minor=$(echo "1.30" | awk -F. '{print $2}') \ --dynamic-fact openshiftVersion.Major=$(echo "4.17" | awk -F. '{print $1}') \ --dynamic-fact openshiftVersion.Minor=$(echo "4.17" | awk -F. '{print $2}')
This commodore
call requires Commodore v1.5.0 or newer. Please make sure to update your local installation.
Configure the OpenShift Installer
-
Generate SSH key
We generate a unique SSH key pair for the cluster as this gives us troubleshooting access.
SSH_PRIVATE_KEY="$(pwd)/ssh_$CLUSTER_ID" export SSH_PUBLIC_KEY="${SSH_PRIVATE_KEY}.pub" ssh-keygen -C "vault@$CLUSTER_ID" -t ed25519 -f $SSH_PRIVATE_KEY -N '' BASE64_NO_WRAP='base64' if [[ "$OSTYPE" == "linux"* ]]; then BASE64_NO_WRAP='base64 --wrap 0' fi vault kv put clusters/kv/${TENANT_ID}/${CLUSTER_ID}/stackit/ssh \ private_key=$(cat $SSH_PRIVATE_KEY | eval "$BASE64_NO_WRAP") ssh-add $SSH_PRIVATE_KEY
-
Prepare
install-config.yaml
You can add more options to the
install-config.yaml
file. Have a look at the config example for more information.For example, you could change the SDN from a default value to something a customer requests due to some network requirements.
export INSTALLER_DIR="$(pwd)/target" mkdir -p "${INSTALLER_DIR}" cat > "${INSTALLER_DIR}/install-config.yaml" <<EOF apiVersion: v1 metadata: name: ${CLUSTER_ID} (1) baseDomain: ${BASE_DOMAIN} (1) platform: none: {} networking: (2) networkType: Cilium pullSecret: | ${PULL_SECRET} sshKey: "$(cat $SSH_PUBLIC_KEY)" EOF
1 Make sure that the values here match the value of $CLUSTER_DOMAIN
when combined as<metadata.name>.<baseDomain>
. Otherwise, the installation will most likely fail.2 (Optional) Configure non-standard pod and service network here (docs). If setting custom CIDR for the OpenShift networking, the corresponding values should be updated in your Commodore cluster definitions. See Cilium Component Defaults and Parameter Reference. Verify with
less catalog/manifests/cilium/olm/*ciliumconfig.yaml
.
Run the OpenShift Installer
The steps in this section aren’t idempotent and have to be completed uninterrupted in one go. If you have to recreate the install config or any of the generated manifests you need to rerun all of the subsequent steps. |
-
Render install manifests (this will consume the
install-config.yaml
)openshift-install --dir "${INSTALLER_DIR}" \ create manifests
-
If you want to change the default "apps" domain for the cluster:
yq w -i "${INSTALLER_DIR}/manifests/cluster-ingress-02-config.yml" \ spec.domain apps.example.com
-
-
Copy pre-rendered extra machine configs
machineconfigs=catalog/manifests/openshift4-nodes/10_machineconfigs.yaml if [ -f $machineconfigs ]; then yq --no-doc -s \ "\"${INSTALLER_DIR}/openshift/99x_openshift-machineconfig_\" + .metadata.name" \ $machineconfigs fi
-
Copy pre-rendered Cilium manifests
cp catalog/manifests/cilium/olm/* ${INSTALLER_DIR}/manifests/
-
Verify that the generated cluster domain matches the desired cluster domain
GEN_CLUSTER_DOMAIN=$(yq e '.spec.baseDomain' \ "${INSTALLER_DIR}/manifests/cluster-dns-02-config.yml") if [ "$GEN_CLUSTER_DOMAIN" != "$CLUSTER_DOMAIN" ]; then echo -e "\033[0;31mGenerated cluster domain doesn't match expected cluster domain: Got '$GEN_CLUSTER_DOMAIN', want '$CLUSTER_DOMAIN'\033[0;0m" else echo -e "\033[0;32mGenerated cluster domain matches expected cluster domain.\033[0;0m" fi
-
Prepare install manifests and ignition config
openshift-install --dir "${INSTALLER_DIR}" \ create ignition-configs
-
Upload ignition config
mc cp "${INSTALLER_DIR}/bootstrap.ign" "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition/" export TF_VAR_ignition_bootstrap=$(mc share download \ --json --expire=4h \ "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition/bootstrap.ign" | jq -r '.share')
Terraform Cluster Config
-
Switch to the tenant repo
pushd "inventory/classes/${TENANT_ID}/"
-
Include no-opsgenie class to prevent monitoring noise during cluster setup
yq eval -i '.classes += "global.distribution.openshift4.no-opsgenie"' ${CLUSTER_ID}.yml;
-
Update cluster config
yq eval -i ".parameters.openshift.infraID = \"$(jq -r .infraID "${INSTALLER_DIR}/metadata.json")\"" \ ${CLUSTER_ID}.yml yq eval -i ".parameters.openshift.clusterID = \"$(jq -r .clusterID "${INSTALLER_DIR}/metadata.json")\"" \ ${CLUSTER_ID}.yml yq eval -i ".parameters.openshift.ssh_key = \"$(cat ${SSH_PUBLIC_KEY})\"" \ ${CLUSTER_ID}.yml
If you use a custom "apps" domain, make sure to set
parameters.openshift.appsDomain
accordingly.APPS_DOMAIN=your.custom.apps.domain yq eval -i ".parameters.openshift.appsDomain = \"${APPS_DOMAIN}\"" \ ${CLUSTER_ID}.yml
By default, the cluster’s update channel is derived from the cluster’s reported OpenShift version. If you want to use a custom update channel, make sure to set
parameters.openshift4_version.spec.channel
accordingly.# Configure the OpenShift update channel as `fast` yq eval -i ".parameters.openshift4_version.spec.channel = \"fast-{ocp-minor-version}\"" \ ${CLUSTER_ID}.yml
-
Set team responsible for handling Icinga alerts
# use lower case for team name. # e.g. TEAM=aldebaran TEAM=<team-name>
-
Prepare Terraform cluster config
CA_CERT=$(jq -r '.ignition.security.tls.certificateAuthorities[0].source' \ "${INSTALLER_DIR}/master.ign" | \ awk -F ',' '{ print $2 }' | \ base64 --decode) yq eval -i ".parameters.openshift4_terraform.terraform_variables.base_domain = \"${BASE_DOMAIN}\"" \ ${CLUSTER_ID}.yml yq eval -i ".parameters.openshift4_terraform.terraform_variables.ignition_ca = \"${CA_CERT}\"" \ ${CLUSTER_ID}.yml yq eval -i ".parameters.openshift4_terraform.terraform_variables.ssh_key = \"$(cat ${SSH_PUBLIC_KEY})\"" \ ${CLUSTER_ID}.yml
-
Configure STACKIT-specific Terraform variables
yq eval -i ".parameters.openshift4_terraform.terraform_variables.image_id = \"${IMAGE_ID}\"" \ ${CLUSTER_ID}.yml yq eval -i ".parameters.openshift4_terraform.terraform_variables.stackit_project_id = \"${STACKIT_PROJECT_ID}\"" \ ${CLUSTER_ID}.yml
You now have the option to further customize the cluster by editing Please look at the configuration reference for the available options. |
Commit changes and compile cluster catalog
-
Review changes. Have a look at the file
${CLUSTER_ID}.yml
. Override default parameters or add more component configurations as required for your cluster. -
Commit changes
git commit -a -m "Setup cluster ${CLUSTER_ID}" git push popd
-
Compile and push cluster catalog
commodore catalog compile ${CLUSTER_ID} --push -i \ --dynamic-fact kubernetesVersion.major=$(echo "1.30" | awk -F. '{print $1}') \ --dynamic-fact kubernetesVersion.minor=$(echo "1.30" | awk -F. '{print $2}') \ --dynamic-fact openshiftVersion.Major=$(echo "4.17" | awk -F. '{print $1}') \ --dynamic-fact openshiftVersion.Minor=$(echo "4.17" | awk -F. '{print $2}')
This commodore
call requires Commodore v1.5.0 or newer. Please make sure to update your local installation.
Provision Infrastructure
-
Check if a Service Account exists for Terraform
stackit service-account list --project-id "${STACKIT_PROJECT_ID}" -ojson | jq -r '.[] | select (.email | startswith("'"${CLUSTER_ID}"'-terraform")) | .email'
-
If the previous command returns an e-mail address, export it.
export SA_EMAIL="`stackit service-account list --project-id "${STACKIT_PROJECT_ID}" -ojson | jq -r '.[] | select (.email | startswith("'"${CLUSTER_ID}"'-terraform")) | .email'`"
-
If the command above didn’t return an e-mail address, create a new Service Account.
SA_EMAIL="`stackit service-account create --name "${CLUSTER_ID}-terraform" --project-id "${STACKIT_PROJECT_ID}" -o json -y | jq -r .email`" stackit curl https://authorization.api.stackit.cloud/v2/$STACKIT_PROJECT_ID/members --data '{"members":[{"subject":"'"${SA_EMAIL}"'","role":"editor"}],"resourceType":"project"}' -XPATCH
-
-
Create a Service Account Token for Terraform
export STACKIT_SERVICE_ACCOUNT_TOKEN="`stackit service-account token create --email "${SA_EMAIL}" --project-id "${STACKIT_PROJECT_ID}" --ttl-days 3 -o json -y | jq -r .token`"
-
Configure Terraform secrets
cat <<EOF > ./terraform.env STACKIT_SERVICE_ACCOUNT_TOKEN TF_VAR_ignition_bootstrap EOF
The next section assumes that Furthermore, without support for STACKIT in |
-
Setup Terraform
Prepare Terraform execution environment# Set terraform image and tag to be used tf_image=$(\ yq eval ".parameters.openshift4_terraform.images.terraform.image" \ dependencies/openshift4-terraform/class/defaults.yml) tf_tag=$(\ yq eval ".parameters.openshift4_terraform.images.terraform.tag" \ dependencies/openshift4-terraform/class/defaults.yml) # Generate the terraform alias base_dir=$(pwd) alias terraform='touch .terraformrc; docker run -it --rm \ -e REAL_UID=$(id -u) \ -e TF_CLI_CONFIG_FILE=/tf/.terraformrc \ --env-file ${base_dir}/terraform.env \ -w /tf \ -v $(pwd):/tf \ --ulimit memlock=-1 \ "${tf_image}:${tf_tag}" /tf/terraform.sh' export GITLAB_REPOSITORY_URL=$(curl -sH "Authorization: Bearer $(commodore fetch-token)" ${COMMODORE_API_URL}/clusters/${CLUSTER_ID} | jq -r '.gitRepo.url' | sed 's|ssh://||; s|/|:|') export GITLAB_REPOSITORY_NAME=${GITLAB_REPOSITORY_URL##*/} export GITLAB_CATALOG_PROJECT_ID=$(curl -sH "Authorization: Bearer ${GITLAB_TOKEN}" "https://git.vshn.net/api/v4/projects?simple=true&search=${GITLAB_REPOSITORY_NAME/.git}" | jq -r ".[] | select(.ssh_url_to_repo == \"${GITLAB_REPOSITORY_URL}\") | .id") export GITLAB_STATE_URL="https://git.vshn.net/api/v4/projects/${GITLAB_CATALOG_PROJECT_ID}/terraform/state/cluster" pushd catalog/manifests/openshift4-terraform/
Initialize Terraformterraform init \ "-backend-config=address=${GITLAB_STATE_URL}" \ "-backend-config=lock_address=${GITLAB_STATE_URL}/lock" \ "-backend-config=unlock_address=${GITLAB_STATE_URL}/lock" \ "-backend-config=username=${GITLAB_USER}" \ "-backend-config=password=${GITLAB_TOKEN}" \ "-backend-config=lock_method=POST" \ "-backend-config=unlock_method=DELETE" \ "-backend-config=retry_wait_min=5"
-
Deploy bootstrap node
cat > override.tf <<EOF module "cluster" { bootstrap_count = 1 master_count = 0 infra_count = 0 worker_count = 0 additional_worker_groups = {} } EOF terraform apply
-
Set up DNS NS records on parent zone using the data from the Terraform output variable ns_records from the previous step
component-openshift4-terraform
doesn’t currently support the STACKIT cluster Terraform module, so at this time, the Terraform output for NS records isn’t provided. -
Wait for bootstrap API to come up
API_URL=$(yq e '.clusters[0].cluster.server' "${INSTALLER_DIR}/auth/kubeconfig") while ! curl --connect-timeout 1 "${API_URL}/healthz" -k &>/dev/null; do echo -n "." sleep 5 done && echo -e "\nAPI is up"
-
Patch Cilium config to allow control plane bootstrap to succeed
We need to temporarily adjust the Cilium config to not use full kube-proxy replacement, since we currently don’t have a way to disable the initial OpenShift-managed kube-proxy deployment. Additionally, because the stackit Cloud Controller Manager accesses the K8s API via service IP, we need to configure Cilium to provide partial kube-proxy replacement so that the CCM can start and untaint the control plane nodes so that other pods can be scheduled.
export KUBECONFIG="${INSTALLER_DIR}/auth/kubeconfig" while ! kubectl get ciliumconfig -A &>/dev/null; do echo -n "." sleep 2 done && echo -e "\nCiliumConfig CR is present" kubectl patch -n cilium ciliumconfig cilium-enterprise --type=merge \ -p '{ "spec": { "cilium": { "kubeProxyReplacement": "false", "nodePort": { "enabled": true }, "socketLB": { "enabled": true }, "sessionAffinity": true, "externalIPs": { "enabled": true }, "hostPort": { "enabled": true } } } }'
-
Deploy control plane nodes
cat > override.tf <<EOF module "cluster" { bootstrap_count = 1 infra_count = 0 worker_count = 0 additional_worker_groups = {} } EOF terraform apply
-
Wait for bootstrap to complete
openshift-install --dir "${INSTALLER_DIR}" \ wait-for bootstrap-complete --log-level debug
If you’re using a CNI other than Cilium you may need to remove the following taint from the nodes to allow the network to come up:
kubectl taint no --all node.cloudprovider.kubernetes.io/uninitialized:NoSchedule-
Once the bootstrap is complete, taint the master nodes again to ensure that they’re properly initialized by the cloud-controller-manager.
kubectl taint no -l node-role.kubernetes.io/master node.cloudprovider.kubernetes.io/uninitialized=:NoSchedule
-
Remove bootstrap node and provision remaining nodes
rm override.tf terraform apply popd
-
Disable OpenShift kube-proxy deployment and revert Cilium patch
kubectl patch network.operator cluster --type=merge \ -p '{"spec":{"deployKubeProxy":false}}' kubectl -n cilium replace -f catalog/manifests/cilium/olm/cluster-network-07-cilium-ciliumconfig.yaml while ! kubectl -n cilium get cm cilium-config -oyaml | grep 'kube-proxy-replacement: "true"' &>/dev/null; do echo -n "." sleep 2 done && echo -e "\nCilium config updated" kubectl -n cilium rollout restart ds/cilium
-
Approve node certs
# Once CSRs in state Pending show up, approve them # Needs to be run twice, two CSRs for each node need to be approved kubectl get csr -w oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | \ xargs oc adm certificate approve kubectl get nodes
-
Label infra nodes
kubectl get node -ojson | \ jq -r '.items[] | select(.metadata.name | test("infra-")).metadata.name' | \ xargs -I {} kubectl label node {} node-role.kubernetes.io/infra=
-
Label worker nodes
kubectl get node -ojson | \ jq -r '.items[] | select(.metadata.name | test("infra|master|storage-")|not).metadata.name' | \ xargs -I {} kubectl label node {} node-role.kubernetes.io/app=
At this point you may want to add extra labels to the additional worker groups, if there are any. -
Enable proxy protocol on ingress controller
kubectl -n openshift-ingress-operator patch ingresscontroller default --type=json \ -p '[{ "op":"replace", "path":"/spec/endpointPublishingStrategy", "value": {"type": "HostNetwork", "hostNetwork": {"protocol": "PROXY"}} }]'
This step isn’t necessary if you’ve disabled the proxy protocol on the load-balancers manually during setup.
By default, PROXY protocol is enabled through the VSHN Commodore global defaults.
-
Wait for installation to complete
openshift-install --dir ${INSTALLER_DIR} \ wait-for install-complete --log-level debug
-
Create secret with S3 credentials for the registry
oc create secret generic image-registry-private-configuration-user \ --namespace openshift-image-registry \ --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=$(mc config host ls ${CLUSTER_ID} -json | jq -r .accessKey) \ --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=$(mc config host ls ${CLUSTER_ID} -json | jq -r .secretKey)
If the registry S3 credentials are created too long after the initial cluster setup, it’s possible that the
openshift-samples
operator has disabled itself because it couldn’t find a working in-cluster registry.If the samples operator is disabled, no templates and builder images will be available on the cluster.
You can check the samples-operator’s state with the following command:
kubectl get config.samples cluster -ojsonpath='{.spec.managementState}'
If the command returns
Removed
, verify that the in-cluster registry pods are now running, and enable the samples operator again:kubectl patch config.samples cluster -p '{"spec":{"managementState":"Managed"}}'
See the upstream documentation for more details on the samples operator.
Setup acme-dns CNAME records for the cluster
You can skip this section if you’re not using Let’s Encrypt for the cluster’s API and default wildcard certificates. |
-
Extract the acme-dns subdomain for the cluster after
cert-manager
has been deployed via Project Syn.fulldomain=$(kubectl -n syn-cert-manager \ get secret acme-dns-client \ -o jsonpath='{.data.acmedns\.json}' | \ base64 -d | \ jq -r '[.[]][0].fulldomain') echo "$fulldomain"
-
Setup the
_acme-challenge
CNAME records in the cluster’s DNS zoneThe
_acme-challenge
records must be created in the same zone as the cluster’sapi
andapps
records respectively. The snippet below assumes that the cluster is configured to use the default "apps" domain in the cluster’s zone.zone_id="`stackit dns zone list -ojson | jq -r '.[] | select(.dnsName == "'"${CLUSTER_DOMAIN}"'") | .id'`" for cname in "api" "apps"; do stackit dns record-set create --zone-id "$zone_id" --ttl 600 --type CNAME --name "_acme-challenge.${cname}" --record "${fulldomain}." done
Ensure emergency admin access to the cluster
-
Check that emergency credentials were uploaded and are accessible:
emergency-credentials-receive "${CLUSTER_ID}" # Follow the instructions to use the downloaded kubeconfig file
You need to be in the passbolt group
VSHN On-Call
.If the command fails, check if the controller is already deployed, running, and if the credentials are uploaded:
kubectl -n appuio-emergency-credentials-controller get emergencyaccounts.cluster.appuio.io -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.lastTokenCreationTimestamp}{"\n"}{end}'
-
Follow the instructions from
emergency-credentials-receive
to use the downloadedkubeconfig
file.export KUBECONFIG="em-${CLUSTER_ID}" kubectl get nodes oc whoami # should output system:serviceaccount:appuio-emergency-credentials-controller:*
If the Let’s Encrypt certificate for the API isn’t fully provisioned yet, you may need to run the following
yq
to use the emergency kubeconfig:yq -i e '.clusters[0].cluster.insecure-skip-tls-verify = true' "em-${CLUSTER_ID}"
-
Invalidate the 10 year admin kubeconfig.
kubectl -n openshift-config patch cm admin-kubeconfig-client-ca --type=merge -p '{"data": {"ca-bundle.crt": ""}}'
Enable Opsgenie alerting
-
Create the standard silence for alerts that don’t have the
syn
labeloc --as cluster-admin -n openshift-monitoring create job --from=cronjob/silence silence-manual oc wait -n openshift-monitoring --for=condition=complete job/silence-manual oc --as cluster-admin -n openshift-monitoring delete job/silence-manual
-
Check the remaining active alerts and address them where neccessary
kubectl --as=cluster-admin -n openshift-monitoring exec sts/alertmanager-main -- \ amtool --alertmanager.url=http://localhost:9093 alert --active
-
Remove the "no-opsgenie" class from the cluster’s configuration
pushd "inventory/classes/${TENANT_ID}/" yq eval -i 'del(.classes[] | select(. == "*.no-opsgenie"))' ${CLUSTER_ID}.yml git commit -a -m "Enable opsgenie alerting on cluster ${CLUSTER_ID}" git push popd
Finalize installation
-
Delete local config files
rm -r ${INSTALLER_DIR}/
-
Remove bootstrap bucket
mc rm -r --force "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition" mc rb "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition"
Post tasks
VSHN
-
Add the cluster to the maintenance template, if necessary
Generic
-
Do a first maintenance