How we run OpenShift 3 at VSHN

This documents how we actually run our OpenShift 3 product (APPUiO) at VSHN. Make sure to read the Introduction first so you have an understanding about all the concepts.

Scope

From an operational standpoint, there is no difference between APPUiO Public and APPUiO Private (and APPUiO Dedicated are just nodes on a Public cluster), so this document applies to all flavors of OCP3 currently being operated by VSHN.

OpenShift 4 will be managed quite different, so most of what’s described here will not apply for OCP4!

Deploying OpenShift 3

Setting up a new OpenShift 3 cluster consists of roughly the following steps:

  • Provisioning & partitioning servers

  • Install & run Puppet, make sure the correct profile_openshift::{master,node,lb,…​} class is applied to each system

  • Configure Ansible Inventory in Hiera data (see the template for OCP v3.11)

  • Run a bunch of Ansible playbooks

  • Configure some additional components (Monitoring, Control panel, LDAP group sync, …​)

  • Review everything

The process is documented in detail in our wiki under OpenShift Checklist / Cluster Setup.

That checklist seems awfully long! So many manual steps!

Yep, that’s true! Initially it was much shorter, but with more and more features added to our stack, the list grew longer and longer! Were we to re-engineer the process from scratch, we would do it differently…​ In retrospect, one is always smarter.

Configuring OpenShift 3

All aspects of an OpenShift 3 cluster are configured via its Ansible Inventory. To modify it, you have to modify the cluster’s profile_openshift3::ansible_master::host_groups key in Hiera (for example, see APPUiO Public on cloudscale.ch’s Hiera data).

profile_openshift3 reads the cluster’s profile_openshift3::ansible_master::host_groups key in Hiera, extrapolates more variables from the ones already set, and applies default values for variables that aren’t set at all in Hiera data. This allows you to overwrite every variable via Hiera!

The basic workflow for most configuration looks like this:

  • Change configuration in Hiera

  • Run Puppet on the cluster’s Infra host to have Puppet regenerate the Inventory

  • Run the appropriate Ansible playbook

After applying changes to components using openshift-ansible, make sure to run the Mungg postconfig playbook (playbooks/postconfig.yml) to reapply any overwritten VSHN customizations.

You can limit the playbook to only apply relevant configurations using --tags. Use --list-tags to see all available tags.

You can find a list of most variables for openshift-ansible in the official OpenShift docs under Configuring Your Inventory File.

The docs don’t document ALL available variables! So unfortunately configuring openshift-ansible requires a lot of source code reading.

openshift-ansible ships a total of 96 different playbooks at the time of writing. In the next few sections we try to give a bit of guidance about WHAT playbook you’ll have to run to roll out a change.

Control Plane

Mostly variables under openshift_master_*.

Common configurations include:

  • authentication providers

  • deployment type (OKD vs OCP) (only OCP is supported at VSHN today)

  • network configuration (cluster, host subnet and service IP ranges)

  • various cluster-wide configurations (default node selector, egress and ingress IP ranges)

With the exception of egress and ingress IP ranges, most IP ranges can’t be changed once the cluster is up and running!

Most used playbooks:

  • openshift-master/config.yml - roll out changes to the control plane components (API, Controllers, Scheduler)

  • openshift-etcd/config.yml - roll out changes to etcd

Nodes

Most configuration of nodes happens via so-called "node groups":

  • each node is part of a node group

  • each node has various "edits"[1] defined in the inventory

  • openshift-ansible creates a ConfigMap in the openshift-node namespace on the cluster

  • openshift-ansible applies each "edit" to said ConfigMap

  • each node runs a sync pod (also in the openshift-node namespace) that keeps the local Kubelet configuration (/etc/origin/node/node-config.yaml) in sync with the appropriate ConfigMap

The way this works (especially the "edits") means that configurations that are removed from the Inventory will stay in the ConfigMap - you’ll have to remove them manually, or set them to something else entirely. On the flip side this means that if you add a configuration to the ConfigMap that’s NOT managed by an edit, it will stay that way.
  • openshift-master/openshift_node_group.yml - Create/update the node group ConfigMaps

  • openshift-node/scaleup.yml - Add new nodes to the cluster (see Adding Hosts to an Existing Cluster)

HAProxy Router

Also known as "OpenShift hosted Router" — variables under openshift_hosted_router_*.

Commonly used playbooks:

  • openshift-hosted/deploy_router.yml - Deploy and reconfigure the router

  • openshift-hosted/redeploy-router-certificates.yml - Update/renew the router certificates. Used to update the default Router wildcard certificate

Setting router environment variables

openshift_hosted_router_edits:
  - key: spec.template.spec.containers[0].env
    action: append
    value:
      name: ROUTER_SLOWLORIS_HTTP_KEEPALIVE
      value: 120s
For some settings, there are inventory variables. For example you can set mungg_router_enable_http2, which will then be applied to the ROUTER_ENABLE_HTTP2 value. Check profile_openshift3 first, it might save you some work!

Docker Registry

Also known as the "OpenShift Hosted Registry" — variables under openshift_hosted_registry_*

Commonly used playbooks:

  • openshift-hosted/deploy_registry.yml - Deploy and reconfigure the registry

Logging

Deploy and configure the OpenShift 3 EFK stack. Variables under openshift_logging_{fluentd,es,kibana}_*.

Commonly used playbooks:

  • openshift-logging/config.yml - Deploy and reconfigure the logging stack

Cluster Monitoring

Deploy and configure Prometheus Cluster Monitoring. This actually comes in two parts:

  1. the openshift-ansible playbooks are used to deploy the Prometheus Operator and configure the main components (Prometheus, Alertmanager, Grafana)

  2. our custom default configuration (ServiceMonitor and PrometheusRule objects) are defined in mungg-cluster-monitoring

  3. additional per-cluster ServiceMonitor and PrometheusRule objects can be created via mungg-cluster-objects

Commonly used playbooks:

  • openshift-monitoring/config.yml - Deploy and reconfigure the Prometheus monitoring stack.

See OpenShift 3 Monitoring for a more in-depth description.


1. patches to its configuration, for example default labels or kubelet arguments. See Configuring Node Group Definitions in the OpenShift docs for more information