Installation on Exoscale

Steps to install an OpenShift 4 cluster on Exoscale.

These steps follow the Installing a cluster on bare metal docs to set up a user provisioned installation (UPI). Terraform is used to provision the cloud infrastructure.

This how-to guide is still a work in progress and will change. It’s currently very specific to VSHN and needs further changes to be more generic.

Prerequisites

Cluster Installation

  1. Register the new OpenShift 4 cluster in Lieutenant: control.vshn.net/syn/lieutenantclusters

  2. Configure input

    export EXOSCALE_ACCOUNT=<exoscale-account>
export EXOSCALE_API_KEY=<exoscale-key>
export EXOSCALE_API_SECRET=<exoscale-secret>
export EXOSCALE_REGION=ch-dk-2
export CLUSTER_ID=<cluster-name>
export BASE_DOMAIN=ocp4-poc.appuio-beta.ch
export PULL_SECRET=<redhat-pull-secret> # From https://cloud.redhat.com/openshift/install/pull-secret

export AWS_ACCESS_KEY_ID=${EXOSCALE_API_KEY}
export AWS_SECRET_ACCESS_KEY=${EXOSCALE_API_SECRET}
export AWS_S3_ENDPOINT="sos-${EXOSCALE_REGION}.exo.io"

  3. Prepare Exoscale CLI

    mkdir -p ~/.exoscale
cat <<EOF >> ~/.exoscale/exoscale.toml

[[accounts]]
  account = "${EXOSCALE_ACCOUNT}"
  defaultZone = "${EXOSCALE_REGION}"
  endpoint = "https://api.exoscale.ch/v1"
  name = "${CLUSTER_ID}"
EOF

  4. Create S3 buckets

    exo sos create "${CLUSTER_ID}-bootstrap-ignition"
exo sos create "${CLUSTER_ID}-tf-state"

  5. Prepare install-config.yaml

    mkdir ${CLUSTER_ID}

cat > "${CLUSTER_ID}/install-config.yaml" <<EOF
apiVersion: v1
metadata:
  name: ${CLUSTER_ID}
baseDomain: ${BASE_DOMAIN}
platform:
  none: {}
pullSecret: |
  ${PULL_SECRET}
sshKey: "$(cat ~/.ssh/id_ed25519.pub)"
EOF

  6. Prepare install manifests and ignition config

    openshift-install --dir ${CLUSTER_ID} \
  create manifests

openshift-install --dir ${CLUSTER_ID} \
  create ignition-configs

for file in bootstrap master worker; do \
  exo sos upload "${CLUSTER_ID}-bootstrap-ignition" "${CLUSTER_ID}/${file}.ign" && \
  exo sos acl add "${CLUSTER_ID}-bootstrap-ignition" "${file}.ign" \
      --public-read ; \
done

  7. Prepare RHCOS images

    RHCOS_VERSION="4.5.2"

curl "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.5/latest/rhcos-${RHCOS_VERSION}-x86_64-openstack.x86_64.qcow2.gz" | gunzip > rhcos-${RHCOS_VERSION}.qcow2

virt-edit -a rhcos-${RHCOS_VERSION}.qcow2 \
  -m /dev/sda1:/ /loader/entries/ostree-1-rhcos.conf \
  -e 's/openstack/metal ignition.config.url=http://169.254.169.254/1.0/user-data/'

exo sos upload "${CLUSTER_ID}-bootstrap-ignition" rhcos-${RHCOS_VERSION}.qcow2
exo sos acl add "${CLUSTER_ID}-bootstrap-ignition" rhcos-${RHCOS_VERSION}.qcow2 --public-read

exo vm template register \
  --checksum $(md5sum rhcos-${RHCOS_VERSION}.qcow2 | awk '{ print $1 }') \
  --boot-mode uefi \
  --disable-password \
  --username core \
  --description "Red Hat Enterprise Linux CoreOS (RHCOS) ${RHCOS_VERSION}" \
  --name "rhcos-${RHCOS_VERSION}" \
  --url "https://${AWS_S3_ENDPOINT}/${CLUSTER_ID}-bootstrap-ignition/rhcos-${RHCOS_VERSION}.qcow2"

export TF_VAR_rhcos_template="rhcos-${RHCOS_VERSION}"

  8. Initialize Terraform

    terraform init \
  -backend-config "bucket=${CLUSTER_ID}-tf-state"

export TF_VAR_cluster_id=$CLUSTER_ID
export TF_VAR_ssh_key="$(cat ~/.ssh/id_rsa.pub)"

  9. Set up DNS NS records on parent zone (see Terraform output ns_records)

    terraform apply \
  -target exoscale_domain_record.exo_nameservers

  10. Bootstrap cluster

    terraform apply \
  -var bootstrap_count=1 \
  -var worker_count=0

  11. Wait for bootstrap to complete

    openshift-install --dir ${CLUSTER_ID} \
  wait-for bootstrap-complete

  12. Remove bootstrap node and provision worker nodes

    terraform apply

  13. Approve worker certs

    export KUBECONFIG=${CLUSTER_ID}/auth/kubeconfig

# Once CSRs in state Pending show up, approve them
# Needs to be run twice, two CSRs for each node need to be approved
kubectl get csr -w
oc get csr -o go-template='{{range .items}}{{if not .status}}{{.metadata.name}}{{"\n"}}{{end}}{{end}}' | \
  xargs oc adm certificate approve

kubectl get nodes

openshift-install --dir ${CLUSTER_ID} \
  wait-for install-complete

  14. Create secret with S3 credentials for the registry (will be automated)

    oc create secret generic image-registry-private-configuration-user \
--namespace openshift-image-registry \
--from-literal=REGISTRY_STORAGE_S3_ACCESSKEY=${AWS_ACCESS_KEY_ID} \
--from-literal=REGISTRY_STORAGE_S3_SECRETKEY=${AWS_SECRET_ACCESS_KEY}

  15. Create wildcard cert for router

    kubectl get secret router-certs-default \
  -n openshift-ingress \
  -ojson --export | \
    jq 'del(.metadata.ownerReferences) | .metadata.name = "router-certs-snakeoil"' | \
  kubectl -n openshift-ingress apply -f -

  16. Make the cluster Project Syn enabled

    Install Steward on the cluster according to wiki.vshn.net/x/ngMBCg

    cat ${CLUSTER_ID}/metadata.json