Cluster Access Tooling

The user interacts with the cluster access tooling through a command line interface.

Routing information is queried from our central inventory, which is managed using Project Syn and can be queried through the Lieutenant API.

See Routing information retrieval for more information.

The tooling uses the user’s SSH configuration. SSH Jumphost (sshop) configuration is included in the user SSH configuration.

The tooling reads the user’s SSH configuration to determine how to connect to the jump hosts and clusters.

The tooling sets up a local SOCKS5 proxy listening on localhost.

The SOCKS5 proxy is transparent to the user, non-routed traffic is directly connected to the destination, while traffic to cluster endpoints is routed through the proxy. Direct connections shouldn’t affect speed or battery life, as they should be handed off to the kernel without going through the proxy.

The proxy can stay running in the background.

We decided to not try to deal with VPNs. There is no stable cross-platform solution for managing the VSHN VPN.

Clusters that require VPN access are routed through an additional VSHN management jump host that has access to the VPN, so users don’t have to worry about setting up the VPN themselves.

Cluster authentication isn’t handled by the cluster access tooling, but the tooling should work well with our existing cluster authentication solutions.

The tool sets up a local SOCKS5 proxy on localhost that routes traffic to cluster endpoints through the appropriate jump hosts based on the routing information retrieved from the Lieutenant API. All other traffic is directly connected to the destination. The implementation should hand off non-routed traffic to the kernel to avoid unnecessary overhead.

The list of routed endpoints should be searched by suffix so that subdomains of cluster endpoints are also routed through the proxy without needing to list every possible subdomain in the routing information. The list should be searched by longest suffix first to allow for more specific routing rules to take precedence over more general ones.

Exclusion rules take precedence over inclusion rules, so if a domain matches both an inclusion and an exclusion rule, it should be routed directly to the destination instead of through the proxy.

A POC implementation of the SOCKS5 was able to demonstrate non-noticeable overhead for new TCP connections, and zero overhead for established connections.

An optional username and password can be set for the SOCKS5 proxy to prevent unauthorized access to the proxy. We don’t see it as necessary to add authentication by default, as the proxy is only listening on localhost, and only does tunneling to cluster endpoints, so it shouldn’t be a security risk.

SSH connections are set up when first required and kept alive as long as the proxy is running to avoid the overhead of setting up new SSH connections for every request. Connections are monitored and automatically re-established if they drop due to network issues or if the machine goes to sleep.

The tooling uses the golang.org/x/crypto/ssh package to set up SSH tunnels to the jump hosts and clusters. This allows us to have full control over the tunnels, we don’t have to worry about managing ssh processes and ports, and we can easily keep the tunnels alive in the background. VSHN SSH configuration is pretty standardized, so we can implement support for the features we use without needing to support every possible SSH configuration option. If a user has a complex SSH configuration that the tooling doesn’t support, we can throw an error early to avoid unexpected behavior.

We evaluated using the ssh command line tool with -D for dynamic port forwarding. The main advantage of using the ssh command line tool is that it can use the user’s existing SSH configuration without having to re-implement all the SSH features in Go. The main advantage of supporting all features is also the biggest disadvantage, as it can lead to unexpected behavior if the user has a complex SSH configuration that the tooling doesn’t support. Another issue is the port management, as each ssh -D command requires a free local port, which can lead to conflicts and makes it harder to manage the proxies. Keeping the tunnel alive through the command line tool can be difficult, especially with ControlMaster and ControlPersist options.

The Go SSH tooling allows to create SSH (agent) servers and clients and we easily end-to-end test the SSH tunneling functionality without needing to set up actual jump hosts and clusters.

A full POC has been implemented using the Go SSH tooling and was able to connect to all our managed clusters with up to three jumps through jump hosts. All SSH features we use in our VSHN SSH configuration are supported, and the implementation feels stable, even when changing networks or under load, and performs well.

Routing information is queried from our central inventory, which is managed using Project Syn and can be queried through the Lieutenant API.

Clusters that need a jump host to access have an additional cluster fact that specifies the jump host to use for accessing the cluster.

Information downloaded from the Lieutenant API is cached locally to allow the tooling to continue working even if the Lieutenant API is temporarily unavailable. New information is downloaded using the update sub-command. The information isn’t expected to change frequently, and we don’t set any TTLs.

Only the last target jump host is specified in the cluster fact, the tooling looks up the routing information for the jump host by querying the SSH configuration. The tooling can support an arbitrary number of jump hosts in the chain without needing to specify them in the routing information.

The following SSH options should be supported to specify jump host chains:

The tooling authenticates to the Lieutenant API using OIDC with the same flow as commodore fetch-token.

The proxy reloads the routing information and invalidates caches on SIGHUP.

The tooling doesn’t implement any special demonization logic, we use standard process management tools to run the proxy in the background, such as systemd on Linux and launchd on macOS.