Getting Started

Basic Overview

Submariner consists of several main components that work in conjunction to securely connect workloads across multiple Kubernetes clusters. For more information about Submariner’s architecture, please refer to the Architecture section.

The Broker

The Broker is an API to which all participating clusters are given access to, and where two objects are exchanged via CRDs:

  • Cluster(.submariner.io): defines a participating cluster and its IP CIDRs.
  • Endpoint(.submariner.io): defines a connection endpoint to a cluster, and the reachable cluster IPs from the endpoint.

The Broker must be deployed on a single Kubernetes cluster. This cluster’s API server must be reachable by all Kubernetes clusters connected by Submariner. It can be a dedicated cluster, or one of the connected clusters.

The Submariner Deployment on a Cluster

Once Submariner is deployed on a cluster with the proper credentials to the Broker it will exchange Cluster and Endpoint objects with other clusters (via push/pull/watching), and start forming connections and routes to other clusters.

Prerequisites

Submariner has a few requirements to get started:

  • At least two Kubernetes clusters, one of which is designated to serve as the central Broker that is accessible by all of your connected clusters; this can be one of your connected clusters, or a dedicated cluster.
  • Minimum supported Kubernetes version is 1.17.
  • Non-overlapping Pod and Service CIDRs between clusters. This is to prevent routing conflicts. For cases where addresses do overlap, Globalnet can be set up.
  • IP reachability between the gateway nodes. When connecting two clusters, at least one of the clusters should have a publicly routable IP address designated to the Gateway node. This is needed for creating the IPsec tunnel between the clusters. The default ports used by IPsec are 4500/UDP and 500/UDP. For clusters behind corporate firewalls that block the default ports, Submariner also supports NAT Traversal (NAT-T) with the option to set custom non-standard ports like 4501/UDP and 501/UDP.
  • Submariner uses UDP port 4800 to encapsulate Pod traffic from worker and master nodes to the Gateway nodes. This is required in order to preserve the source IP addresses of the Pods. Ensure that firewall configuration allows 4800/UDP across all nodes in the cluster in both directions.
  • Submariner uses TCP port 8080 to export metrics on the Gateway nodes. Ensure that firewall configuration allows ingress 8080/TCP on the Gateway nodes so that other nodes in the cluster can access it. Also, no other workload on the Gateway nodes should be listening on TCP port 8080.
  • Worker node IPs on all connected clusters must be outside of the Pod/Service CIDR ranges.

An example of three clusters configured to use with Submariner (without Globalnet) would look like the following:

Cluster Name Provider Pod CIDR Service CIDR Cluster Nodes CIDR
broker AWS 10.42.0.0/16 10.43.0.0/16 192.168.1.0/24
west vSphere 10.0.0.0/16 10.1.0.0/16 192.168.1.0/24
east On-Prem 10.98.0.0/16 10.99.0.0/16 192.168.1.0/24

Support Matrix

Submariner is designed to be cloud provider agnostic, and should run in any standard Kubernetes cluster. Presently, Submariner has been tested with the following network (CNI) Plugins that leverage kube-proxy with iptables mode:

Deployment

The available methods for deployment are:

subctl greatly simplifies the deployment of Submariner, and is therefore the recommended deployment method.