APPUiO Managed OpenShift 4 on Exoscale

Architecture overview

The APPUiO Managed OpenShift 4 architecture on Exoscale is based on the generic APPUiO Managed OpenShift 4 architecture. We expect that readers of this document are familiar with the generic APPUiO Managed OpenShift 4 architecture and the overall Kubernetes and OpenShift 4 architecture.

This architecture document extends upon the generic architecture by defining how the APPUiO Managed OpenShift 4 cluster is embedded into the Exoscale environment. The diagram below shows a detailed view of how an APPUiO Managed OpenShift 4 cluster is embedded into the Exoscale environment.

OCP4 Exoscale architecture
Figure 1. APPUiO Managed OpenShift4 on Exoscale architecture

The following sections of this document provide detailed descriptions for the elements shown in the architecture diagram.

Exoscale requirements

APPUiO Managed OpenShift 4 on Exoscale needs a Load Balancer setup that must meet the following requirements:

  1. API load balancer: Provides a common endpoint to interact with OpenShift and Kubernetes.

  2. Ingress load balancer: Provides an endpoint for application traffic flowing in from outside the cluster.

See the upstream documentation for details on Exoscale requirements.

Networking

Security Groups

On Exoscale, APPUiO Managed OpenShift 4 uses public IPs for each node in the cluster. See Limitations of the Exoscale environment.

The individual VMs are placed in Security Groups to restrict access and isolate the nodes from the public internet.

On the Exoscale environment there is no single stable egress IP. Every node uses a dynamic public IP for egress traffic, which it’s not suited for any forms of whitelisting.

Virtual IPs

To expose applications and the Kubernetes API outside the cluster, APPUiO Managed OpenShift 4 manages two floating IPs:

  1. The "API VIP" for the Kubernetes and OpenShift API. APPUiO Managed OpenShift 4 uses a public floating IP as the API VIP.

  2. The "Ingress VIP" for the OpenShift Ingress Router. APPUiO Managed OpenShift 4 uses a public floating IP as the Ingress VIP.

APPUiO Managed OpenShift 4 uses two Load Balancer instances to manage the API and ingress VIPs and distributes traffic to the master / infrastructure nodes.

Pod and service networks

APPUiO Managed Openshift 4 uses Cilium to provide in-cluster networking. Cilium allocates two cluster-internal networks:

  1. The pod network: every pod on the cluster will get an IP address from this network. This network enables basic in-cluster connectivity. APPUiO Managed OpenShift 4 uses 10.128.0.0/14 as the pod network. Each node in the cluster is assigned a /23 from this range. Pods on a node are always assigned an IP from the range allocated to that node.

  2. Service network: used for service discovery. Traffic to IPs in this network is forwarded to the appropriate pods by Cilium. APPUiO Managed OpenShift 4 uses 172.30.0.0/16 as the service network.

Both of these networks are interanl to the OpenShift 4 cluster. Therefore, the IP CIDRs for these networks must not be routable from the outside. Additionally, the same IP CIDRs can be reused for multiple OpenShift 4 clusters.

However, the chosen CIDRs shouldn’t overlap with existing networks allocated by the customer. If there are overlaps, external systems in the overlapping ranges won’t be accessible from within the OpenShift 4 cluster. The pod and service network CIDRs can be customized if and only if there are conflicts.

Exposing the cluster

We provide a CNAME target record to point additional DNS records to.

External services

APPUiO Managed OpenShift 4 requires various external services.

VSHN services

APPUiO Managed OpenShift 4 requires access to VSHN’s Project Syn infrastructure. The Project Syn infrastructure components that must be reachable are

  • the Project Syn API at https://api.syn.vshn.net

  • the Project Syn Vault at https://vault-prod.syn.vshn.net

  • VSHN’s GitLab instance at ssh://git@git.vshn.net

  • VSHN’s acme-dns instance at https://acme-dns-api.vshn.net

Additionally, APPUiO Managed OpenShift 4 requires access to VSHN’s identity management:

  • VSHN SSO at https://id.vshn.net

Finally, APPUiO Managed OpenShift 4 requires access to VSHN’s central metrics storage at https://metrics-receive.appuio.net

Red Hat services

See the upstream documentation for the full list of services.

The most important services for APPUiO Managed OpenShift 4 are

  • the Red Hat container registries at registry.redhat.io and registry.access.redhat.com.

  • the OpenShift Update Service (OSUS) at https://api.openshift.com.

3rd party services

Finally, APPUiO Managed OpenShift 4 requires access to a number of third party services:

  • OpsGenie at https://api.opsgenie.com

  • Passbolt at https://cloud.passbolt.com/vshn

  • Let’s Encrypt at https://acme-v02.api.letsencrypt.com and https://acme-staging-v02.api.letsencrypt.com

  • Various container registries

    • GitHub at ghcr.io

    • Quay at quay.io

    • DockerHub at docker.io

    • Google container registry at gcr.io

    • Kubernetes container registry at registry.k8s.io

Storage

APPUiO managed OpenShift 4 requires 3 different types of storage:

  1. Root disks

  2. Persistent volumes

  3. S3 compatible object storage

Root disks

Root disks are virtual block devices (100 GiB) which are attached to the VMs which make up the APPUiO Managed OpenShift 4 cluster. The root disks are allocated and attached to the VM when the VM is created. They hold the operating system and temporary data. They’re ephemeral (no application data is stored on them), and don’t need to be backed up. Finally, root disks are deleted when the VM to which they’re attached is deleted.

Persistent volumes

Persistent volumes are virtual block devices with arbitrary sizes. They’re allocated dynamically based on requests from workloads (applications or infrastructure components) within the cluster. These block devices are automatically attached to the VM hosting the application container. They’re deleted when the corresponding Kubernetes PersistentVolume resource is deleted.

Exoscale does not provide storage usable by Kubernetes as persistent volumes. To fill this gap, {product} in {intra-type} uses APPUiO Managed Storage Cluster to provide storage to be used as read write once and read write many persistent volumes.

These devices hold application data, but backups are usually done from within the cluster.

S3 compatible object storage

Various OpenShift components, such as the integrated image registry, the logging stack and backups, require S3 compatible object storage.

Glossary

Components Exoscale

Name Description provided by

Security Group

Exoscale Security Groups provide a modular way to define and compose firewall rules.

Security Groups hold two different types of information: * A list of rules to apply to traffic * A list of member instances in the security group which allows using groups as traffic sources or destinations in rules

See Upstream Documentation.

Exoscale

S3 compatible storage

Various OpenShift components require S3 compatible storage. This storage is provided by Exoscale.

The main APPUiO Managed OpenShift 4 components that use object storage are

  • OpenShift integrated image registry

  • OpenShift logging stack

  • APPUiO Managed cluster backups

Exoscale

Components General

Name Description provided by

Installer

A CLI tool that bootstraps an OpenShift 4 cluster based on a configuration file.

VSHN / Red Hat

Bootstrap Node

A temporary VM in the cluster machine network which is provisioned by the installer to facilitate the initial setup of the cluster. This VM is decommissioned by the installer once the cluster installation is completed.

VSHN / Installer

Pod network

A subnet that’s internal to the Openshift 4 cluster. This subnet shouldn’t be routable from outside the cluster.

This subnet is managed by Cilium and is implemented with VXLAN traffic between the cluster VMs

APPUiO Managed OpenShift 4 uses 10.128.0.0/14 as the pod network. If the pod network IP range conflicts with existing subnets, the pod network IP range can be adjusted.

VSHN / Cilium

Service network

A subnet that’s internal to the OpenShift 4 cluster. This subnet shouldn’t be routable from outside the cluster.

This subnet is managed by Cilium and is implemented with eBPF rules on the cluster VMs.

APPUiO Managed OpenShift 4 uses 172.30.0.0/16 as the service network. If the service network IP range conflicts with existing subnets, the service network IP range can be adjusted.

VSHN / Cilium

DNS

The APPUiO Managed OpenShift 4 cluster’s base DNS records are defined and managed by VSHN. All records must be publicly resolvable. To expose applications under a customer domain, a CNAME target is provided.

VSHN

Storage Cluster

The APPUiO Managed Storage Cluster offers advanced cloud-native storage capabilities for APPUiO Managed OpenShift 4.

This product is based on Rook and uses Ceph as it’s underlying storage technology.

See APPUiO Managed Storage Cluster product page for more details.

VSHN / Rook

Other terms

Name Description

Node

A virtual machine that’s part of an OpenShift 4 cluster

Control plane

A collection of components that

  • facilitate the management of the container platform

  • manage the virtual hardware making up the cluster

  • manage the applications running on the cluster