Installation on Exoscale

Steps to install an OpenShift 4 cluster on Exoscale.

These steps follow the Installing a cluster on bare metal docs to set up a user provisioned installation (UPI). Terraform is used to provision the cloud infrastructure.

This how-to guide is still a work in progress and will change. It’s currently very specific to VSHN and needs further changes to be more generic.


Cluster Installation

  1. Register the new OpenShift 4 cluster in Lieutenant:

  2. Configure input

    export EXOSCALE_ACCOUNT=<exoscale-account>
    export EXOSCALE_API_KEY=<exoscale-key>
    export EXOSCALE_API_SECRET=<exoscale-secret>
    export EXOSCALE_REGION=ch-dk-2
    export CLUSTER_ID=<cluster-name>
    export PULL_SECRET=<redhat-pull-secret> # From
  3. Prepare Exoscale CLI

    mkdir -p ~/.exoscale
    cat <<EOF >> ~/.exoscale/exoscale.toml
      account = "${EXOSCALE_ACCOUNT}"
      defaultZone = "${EXOSCALE_REGION}"
      endpoint = ""
      name = "${CLUSTER_ID}"
  4. Create S3 buckets

    exo sos create "${CLUSTER_ID}-bootstrap-ignition"
    exo sos create "${CLUSTER_ID}-tf-state"
  5. Prepare install-config.yaml

    mkdir ${CLUSTER_ID}
    cat > "${CLUSTER_ID}/install-config.yaml" <<EOF
    apiVersion: v1
      name: ${CLUSTER_ID}
    baseDomain: ${BASE_DOMAIN}
      none: {}
    pullSecret: |
    sshKey: "$(cat ~/.ssh/"
  6. Prepare install manifests and ignition config

    openshift-install --dir ${CLUSTER_ID} \
      create manifests
    openshift-install --dir ${CLUSTER_ID} \
      create ignition-configs
    for file in bootstrap master worker; do \
      exo sos upload "${CLUSTER_ID}-bootstrap-ignition" "${CLUSTER_ID}/${file}.ign" && \
      exo sos acl add "${CLUSTER_ID}-bootstrap-ignition" "${file}.ign" \
          --public-read ; \
  7. Prepare RHCOS images

    curl "${RHCOS_VERSION}-x86_64-openstack.x86_64.qcow2.gz" | gunzip > rhcos-${RHCOS_VERSION}.qcow2
    virt-edit -a rhcos-${RHCOS_VERSION}.qcow2 \
      -m /dev/sda1:/ /loader/entries/ostree-1-rhcos.conf \
      -e 's/openstack/metal ignition.config.url='
    exo sos upload "${CLUSTER_ID}-bootstrap-ignition" rhcos-${RHCOS_VERSION}.qcow2
    exo sos acl add "${CLUSTER_ID}-bootstrap-ignition" rhcos-${RHCOS_VERSION}.qcow2 --public-read
    exo vm template register \
      --checksum $(md5sum rhcos-${RHCOS_VERSION}.qcow2 | awk '{ print $1 }') \
      --boot-mode uefi \
      --disable-password \
      --username core \
      --description "Red Hat Enterprise Linux CoreOS (RHCOS) ${RHCOS_VERSION}" \
      --name "rhcos-${RHCOS_VERSION}" \
      --url "https://${AWS_S3_ENDPOINT}/${CLUSTER_ID}-bootstrap-ignition/rhcos-${RHCOS_VERSION}.qcow2"
    export TF_VAR_rhcos_template="rhcos-${RHCOS_VERSION}"
  8. Initialize Terraform

    terraform init \
      -backend-config "bucket=${CLUSTER_ID}-tf-state"
    export TF_VAR_cluster_id=$CLUSTER_ID
    export TF_VAR_ssh_key="$(cat ~/.ssh/"
  9. Set up DNS NS records on parent zone (see Terraform output ns_records)

    terraform apply \
      -target exoscale_domain_record.exo_nameservers
  10. Bootstrap cluster

    terraform apply \
      -var bootstrap_count=1 \
      -var worker_count=0
  11. Wait for bootstrap to complete

    openshift-install --dir ${CLUSTER_ID} \
      wait-for bootstrap-complete
  12. Remove bootstrap node and provision worker nodes

    terraform apply
  13. Approve worker certs

    export KUBECONFIG=${CLUSTER_ID}/auth/kubeconfig
    # Once CSRs in state Pending show up, approve them
    # Needs to be run twice, two CSRs for each node need to be approved
    kubectl get csr -w
    oc get csr -o go-template='{{range .items}}{{if not .status}}{{}}{{"\n"}}{{end}}{{end}}' | \
      xargs oc adm certificate approve
    kubectl get nodes
    openshift-install --dir ${CLUSTER_ID} \
      wait-for install-complete
  14. Create secret with S3 credentials for the registry (will be automated)

    oc create secret generic image-registry-private-configuration-user \
    --namespace openshift-image-registry \
  15. Create wildcard cert for router

    kubectl get secret router-certs-default \
      -n openshift-ingress \
      -ojson --export | \
        jq 'del(.metadata.ownerReferences) | = "router-certs-snakeoil"' | \
      kubectl -n openshift-ingress apply -f -
  16. Make the cluster Project Syn enabled

    Install Steward on the cluster according to

    cat ${CLUSTER_ID}/metadata.json