Installation on cloudscale.ch
Steps to install an OpenShift 4 cluster on cloudscale.ch.
These steps follow the Installing a cluster on bare metal docs to set up a user provisioned installation (UPI). Terraform is used to provision the cloud infrastructure.
The commands are idempotent and can be retried if any of the steps fail. The certificates created during bootstrap are only valid for 24h. So make sure you complete these steps within 24h. |
This how-to guide is still a work in progress and will change. It’s currently very specific to VSHN and needs further changes to be more generic. |
Starting situation
-
You already have a Tenant and its git repository
-
You have a CCSP Red Hat login and are logged into Red Hat Openshift Cluster Manager
-
You want to register a new cluster in Lieutenant and are about to install Openshift 4 on Cloudscale
Prerequisites
-
docker
-
mc
Minio client (aliased tomc
if necessary) -
jq
-
yq
yq YAML processor (version 4 or higher) -
vault
Vault CLI -
qemu-img
-
curl
-
gzip
Make sure the version of openshift-install and the rhcos image is the same, otherwise ignition will fail. |
Cluster Installation
Register the new OpenShift 4 cluster in Lieutenant.
Set up LDAP service
-
Create an LDAP service
Use control.vshn.net/vshn/services/_create to create a service. The name must contain the customer and the cluster name. And then put the LDAP service ID in the following variable:
export LDAP_ID="Your_LDAP_ID_here" export LDAP_PASSWORD="Your_LDAP_pw_here"
Configure input
# From https://control.cloudscale.ch/user/api-tokens
export CLOUDSCALE_TOKEN=<cloudscale-api-token>
# From https://git.vshn.net/profile/personal_access_tokens
export GITLAB_TOKEN=<gitlab-api-token>
# For example: https://api.syn.vshn.net
# IMPORTANT: do NOT add a trailing `/`. Commands below will fail.
export COMMODORE_API_URL=<lieutenant-api-endpoint>
# See https://wiki.vshn.net/x/ngMBCg#ClusterRegistryinLieutenantSynthesizeaCluster(synfection)-Preparation
export COMMODORE_API_TOKEN=<lieutenant-api-token>
export CLUSTER_ID=<lieutenant-cluster-id> # Looks like: c-<verb>-<noun>-<number>
export TENANT_ID=$(curl -sH "Authorization: Bearer ${COMMODORE_API_TOKEN}" ${COMMODORE_API_URL}/clusters/${CLUSTER_ID} | jq -r .tenant)
export BASE_DOMAIN=appuio-beta.ch
export PULL_SECRET='<redhat-pull-secret>' # As copied from https://cloud.redhat.com/openshift/install/pull-secret "Copy pull secret". value must be inside quotes.
For BASE_DOMAIN
explanation, see DNS Scheme.
Set up S3 bucket
-
Create S3 bucket
-
If a bucket user already exists for this cluster:
# Use already existing bucket user response=$(curl -sH "Authorization: Bearer ${CLOUDSCALE_TOKEN}" \ https://api.cloudscale.ch/v1/objects-users | \ jq -e ".[] | select(.display_name == \"${CLUSTER_ID}\")")
-
To create a new bucket user:
# Create a new user response=$(curl -sH "Authorization: Bearer ${CLOUDSCALE_TOKEN}" \ -F display_name=${CLUSTER_ID} \ https://api.cloudscale.ch/v1/objects-users)
-
-
Configure the Minio client
export REGION=$(curl -sH "Authorization: Bearer ${COMMODORE_API_TOKEN}" ${COMMODORE_API_URL}/clusters/${CLUSTER_ID} | jq -r .facts.region) mc config host add \ "${CLUSTER_ID}" "https://objects.${REGION}.cloudscale.ch" \ $(echo $response | jq -r '.keys[0].access_key') \ $(echo $response | jq -r '.keys[0].secret_key') mc mb --ignore-existing \ "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition"
Upload Red Hat CoreOS image
-
Export the Authorization header for the Cloudscale API.
export AUTH_HEADER="Authorization: Bearer ${CLOUDSCALE_TOKEN}"
The variable
CLOUDSCALE_TOKEN
could be used directly. Exporting the variableAUTH_HEADER
is done to be compatible with the Cloudscale API documentation. -
Check if image already exists
curl -sH "$AUTH_HEADER" https://api.cloudscale.ch/v1/custom-images | jq -r '.[] | select(.slug == "rhcos-4.7") | .href'
If a URL is printed to the output, you can skip the next steps and directly jump to the next section.
-
Fetch and convert the latest Red Hat CoreOS image
curl -sL https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.7/4.7.0/rhcos-4.7.0-x86_64-openstack.x86_64.qcow2.gz | gzip -d > rhcos-4.7.gcow qemu-img convert rhcos-4.7.gcow rhcos-4.7.raw
-
Upload the image to S3 and make it public
mc cp rhcos-4.7.raw "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition/" mc policy set download "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition/rhcos-4.7.raw"
The output of the above looks like an error. But when checking with the following, the result is as expected.
mc policy get "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition/rhcos-4.7.raw"
The output should be
Access permission for `[…]-bootstrap-ignition/rhcos-4.7.raw
isdownload`
-
Import the image to Cloudscale
curl -i -H "$AUTH_HEADER" \ -F url="$(mc share download --json "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition/rhcos-4.7" | jq -r .url)" \ -F name='RHCOS 4.7' \ -F zones=rma1 \ -F slug=rhcos-4.7 \ -F source_format=raw \ -F user_data_handling=pass-through \ https://api.cloudscale.ch/v1/custom-images/import
Set secrets in Vault
export VAULT_ADDR=https://vault-prod.syn.vshn.net
vault login -method=ldap username=<your.name>
# Set the cloudscale.ch access secrets
vault kv put clusters/kv/${TENANT_ID}/${CLUSTER_ID}/cloudscale \
token=${CLOUDSCALE_TOKEN} \
s3_access_key=$(mc config host ls ${CLUSTER_ID} -json | jq -r .accessKey) \
s3_secret_key=$(mc config host ls ${CLUSTER_ID} -json | jq -r .secretKey)
# Generate an HTTP secret for the registry
vault kv put clusters/kv/${TENANT_ID}/${CLUSTER_ID}/registry \
httpSecret=$(LC_ALL=C tr -cd "A-Za-z0-9" </dev/urandom | head -c 128)
# Set the LDAP password
vault kv put clusters/kv/${TENANT_ID}/${CLUSTER_ID}/vshn-ldap \
bindPassword=${LDAP_PASSWORD}
# Generate a master password for K8up backups
vault kv put clusters/kv/${TENANT_ID}/${CLUSTER_ID}/global-backup \
password=$(LC_ALL=C tr -cd "A-Za-z0-9" </dev/urandom | head -c 32)
# Generate a password for the cluster object backups
vault kv put clusters/kv/${TENANT_ID}/${CLUSTER_ID}/cluster-backup \
password=$(LC_ALL=C tr -cd "A-Za-z0-9" </dev/urandom | head -c 32)
# Copy the Dagobert OpenShift Node Collector Credentials
vault kv get -format=json "clusters/kv/template/dagobert" | jq '.data.data' \
| vault kv put -cas=0 "clusters/kv/${TENANT_ID}/${CLUSTER_ID}/dagobert" -
OpenShift Installer Setup
For the following steps, change into a clean directory (for example a directory in your home).
These are the only steps which aren’t idempotent and have to be completed uninterrupted in one go. If you have to recreate the install config or any of the generated manifests you need to rerun all of the subsequent steps. |
You can add more options to the For example, you could change the SDN from a default value to something a customer requests due to some network requirements. |
-
Prepare
install-config.yaml
mkdir -p target cat > target/install-config.yaml <<EOF apiVersion: v1 metadata: name: ${CLUSTER_ID} baseDomain: ${BASE_DOMAIN} platform: none: {} pullSecret: | ${PULL_SECRET} EOF
-
Render install manifests (this will consume the
install-config.yaml
)openshift-install --dir target \ create manifests
-
If you want to change the default "apps" domain for the cluster:
yq w -i target/manifests/cluster-ingress-02-config.yml \ spec.domain apps.example.com
-
-
Render and upload ignition config (this will consume all the manifests)
openshift-install --dir target \ create ignition-configs mc cp target/bootstrap.ign "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition/" export TF_VAR_ignition_bootstrap=$(mc share download \ --json --expire=4h \ "${CLUSTER_ID}/${CLUSTER_ID}-bootstrap-ignition/bootstrap.ign" | jq -r '.share')
Terraform Cluster Config
Check Running Commodore for details on how to run commodore. |
-
Prepare Commodore inventory.
mkdir -p inventory/classes/ git clone $(curl -sH"Authorization: Bearer ${COMMODORE_API_TOKEN}" "${COMMODORE_API_URL}/tenants/${TENANT_ID}" | jq -r '.gitRepo.url') inventory/classes/${TENANT_ID}
-
Prepare Terraform cluster config
CA_CERT=$(jq -r '.ignition.security.tls.certificateAuthorities[0].source' \ target/master.ign | \ awk -F ',' '{ print $2 }' | \ base64 --decode) pushd "inventory/classes/${TENANT_ID}/" yq eval -i '.applications += ["openshift4-cloudscale"]' ${CLUSTER_ID}.yml yq eval -i ".parameters.openshift.infraID = \"$(jq -r .infraID ../../../target/metadata.json)\"" \ ${CLUSTER_ID}.yml yq eval -i ".parameters.openshift.clusterID = \"$(jq -r .clusterID ../../../target/metadata.json)\"" \ ${CLUSTER_ID}.yml yq eval -i ".parameters.openshift.appsDomain = \"apps.${CLUSTER_ID}.${BASE_DOMAIN}\"" \ ${CLUSTER_ID}.yml yq eval -i ".parameters.openshift4_cloudscale.variables.base_domain = \"${BASE_DOMAIN}\"" \ ${CLUSTER_ID}.yml yq eval -i ".parameters.openshift4_cloudscale.variables.ignition_ca = \"${CA_CERT}\"" \ ${CLUSTER_ID}.yml yq eval -i ".parameters.vshnLdap.serviceId = \"${LDAP_ID}\"" \ ${CLUSTER_ID}.yml # Have a look at the file ${CLUSTER_ID}.yml. # Override any default parameters or add more component configuration. git commit -a -m "Setup cluster ${CLUSTER_ID}" git push popd
-
Compile and push Terraform setup
commodore catalog compile ${CLUSTER_ID} --push -i
Provision Infrastructure
-
Setup Terraform
Prepare terraform
# Set terraform image and tag to be used tf_image=$(\ yq eval ".parameters.openshift4_cloudscale.images.terraform.image" \ dependencies/openshift4-cloudscale/class/defaults.yml) tf_tag=$(\ yq eval ".parameters.openshift4_cloudscale.images.terraform.tag" \ dependencies/openshift4-cloudscale/class/defaults.yml) # Generate the terraform alias alias terraform='docker run -it --rm \ -e CLOUDSCALE_TOKEN="${CLOUDSCALE_TOKEN}" \ -e TF_VAR_ignition_bootstrap="${TF_VAR_ignition_bootstrap}" \ -w /tf \ -v $(pwd):/tf \ --ulimit memlock=-1 \ ${tf_image}:${tf_tag} terraform' export GITLAB_REPOSITORY_URL=$(curl -sH "Authorization: Bearer ${COMMODORE_API_TOKEN}" ${COMMODORE_API_URL}/clusters/${CLUSTER_ID} | jq -r '.gitRepo.url' | sed 's|ssh://||; s|/|:|') export GITLAB_REPOSITORY_NAME=${GITLAB_REPOSITORY_URL##*/} export GITLAB_CATALOG_PROJECT_ID=$(curl -sH "Authorization: Bearer ${GITLAB_TOKEN}" "https://git.vshn.net/api/v4/projects?simple=true&search=${GITLAB_REPOSITORY_NAME/.git}" | jq -r ".[] | select(.ssh_url_to_repo == \"${GITLAB_REPOSITORY_URL}\") | .id") export GITLAB_STATE_URL="https://git.vshn.net/api/v4/projects/${GITLAB_CATALOG_PROJECT_ID}/terraform/state/cluster" pushd catalog/manifests/openshift4-cloudscale/
Initiate terraform
terraform init \ "-backend-config=address=${GITLAB_STATE_URL}" \ "-backend-config=lock_address=${GITLAB_STATE_URL}/lock" \ "-backend-config=unlock_address=${GITLAB_STATE_URL}/lock" \ "-backend-config=username=$(whoami)" \ "-backend-config=password=${GITLAB_TOKEN}" \ "-backend-config=lock_method=POST" \ "-backend-config=unlock_method=DELETE" \ "-backend-config=retry_wait_min=5"
-
Provision bootstrap node
cat > override.tf <<EOF module "cluster" { bootstrap_count = 1 master_count = 0 infra_count = 0 worker_count = 0 } EOF terraform apply
-
Create the first shown DNS records
-
Wait for the DNS records to propagate!
sleep 600 host "api.${CLUSTER_ID}.${BASE_DOMAIN}"
-
Provision master nodes
cat > override.tf <<EOF module "cluster" { bootstrap_count = 1 infra_count = 0 worker_count = 0 } EOF terraform apply
-
Add the remaining shown DNS records to the previous ones.
-
Wait for bootstrap to complete
openshift-install --dir ../../../target \ wait-for bootstrap-complete
-
Remove bootstrap node and provision infra nodes
cat > override.tf <<EOF module "cluster" { worker_count = 0 } EOF terraform apply export KUBECONFIG="$(pwd)/../../../target/auth/kubeconfig" # Once CSRs in state Pending show up, approve them # Needs to be run twice, two CSRs for each node need to be approved while sleep 3; do \ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | \ xargs oc adm certificate approve; \ done kubectl get nodes -lnode-role.kubernetes.io/worker kubectl label node -lnode-role.kubernetes.io/worker \ node-role.kubernetes.io/infra=""
-
Wait for installation to complete
openshift-install --dir ../../../target \ wait-for install-complete
-
Provision worker nodes
rm override.tf terraform apply # Once CSRs in state Pending show up, approve them # Needs to be run twice, two CSRs for each node need to be approved while sleep 3; do \ oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | \ xargs oc adm certificate approve; \ done kubectl label --overwrite node -lnode-role.kubernetes.io/worker \ node-role.kubernetes.io/app="" kubectl label node -lnode-role.kubernetes.io/infra \ node-role.kubernetes.io/app-
-
Create secret with S3 credentials for the registry (will be automated)
oc create secret generic image-registry-private-configuration-user \ --namespace openshift-image-registry \ --from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=$(mc config host ls ${CLUSTER_ID} -json | jq -r .accessKey) \ --from-literal=REGISTRY_STORAGE_S3_SECRETKEY=$(mc config host ls ${CLUSTER_ID} -json | jq -r .secretKey)
-
Make the cluster Project Syn enabled
Install Steward on the cluster (see wiki for more details):
export LIEUTENANT_NS="lieutenant-prod" # or lieutenant-[dev,int] accordingly export LIEUTENANT_AUTH="Authorization:Bearer ${COMMODORE_API_TOKEN}" # Reset the token curl \ -H "${LIEUTENANT_AUTH}" \ -H "Content-Type: application/json-patch+json" \ -X PATCH \ -d '[{ "op": "remove", "path": "/status/bootstrapToken" }]' \ "https://rancher.vshn.net/k8s/clusters/c-c6j2w/apis/syn.tools/v1alpha1/namespaces/${LIEUTENANT_NS}/clusters/${CLUSTER_ID}/status" kubectl --kubeconfig target/auth/kubeconfig apply -f $(curl -sH "${LIEUTENANT_AUTH}" "https://${COMMODORE_API_URL}/clusters/${CLUSTER_ID}" | jq -r ".installURL")
-
Save the admin credentials in the password manager. You can find the password in the file
target/auth/kubeadmin-password
and the kubeconfig intarget/auth/kubeconfig
popd ls -l target/auth/
-
Delete local config files
rm -r target/