Getting Started

Basic Overview

Submariner consists of several main components that work in conjunction to securely connect workloads across multiple Kubernetes clusters. For more information about Submariner’s architecture, please refer to the Architecture section.

The Broker

The Broker is an API that all participating clusters are given access to, and where two objects are exchanged via CRDs in .submariner.io:

• Cluster: defines a participating cluster and its IP CIDRs.
• Endpoint: defines a connection endpoint to a cluster, and the reachable cluster IPs from the endpoint.

The Broker must be deployed on a single Kubernetes cluster. This cluster’s API server must be reachable by all Kubernetes clusters connected by Submariner. It can be a dedicated cluster, or one of the connected clusters.

The Submariner Deployment on a Cluster

Once Submariner is deployed on a cluster with the proper credentials to the Broker it will exchange Cluster and Endpoint objects with other clusters (via push/pull/watching), and start forming connections and routes to other clusters.

Prerequisites

Submariner has a few requirements to get started:

• At least two Kubernetes clusters, one of which is designated to serve as the central Broker that is accessible by all of your connected clusters; this can be one of your connected clusters, or a dedicated cluster.
• The oldest tested Kubernetes version is 1.19. Older versions are known not to work with Submariner. Service discovery requires Kubernetes 1.21 or later.
• Non-overlapping Pod and Service CIDRs between clusters. This is to prevent routing conflicts. For cases where addresses do overlap, Globalnet can be set up.
• IP reachability between the gateway nodes. When connecting two clusters, the gateways must have at least one-way connectivity to each other on their public or private IP address and encapsulation port. This is needed for creating the tunnels between the clusters. The default encapsulation port is 4500/UDP, for NAT Traversal discovery port 4490/UDP is used. For clusters behind corporate firewalls that block the default ports, Submariner also supports NAT Traversal (NAT-T) with the option to set custom non-standard ports like 4501/UDP.
• Submariner uses UDP port 4800 to encapsulate Pod traffic from worker and master nodes to the Gateway nodes. This is required in order to preserve the source IP addresses of the Pods. Ensure that firewall configuration allows 4800/UDP across all nodes in the cluster in both directions. This is not a requirement when using OVN-Kubernetes CNI.
• If the gateway nodes are directly reachable over their private IPs without any NAT in between, ensure that firewall configuration allows ESP protocol on the gateway nodes.
• Worker node IPs on all connected clusters must be outside of the Pod/Service CIDR ranges.
• Submariner can be deployed on x86-64 and ARM64 nodes. (Submariner components are deployed on all nodes in the cluster, so all nodes must be x86-64 or ARM64.)

An example of three clusters configured to use with Submariner (without Globalnet) would look like the following:

systemd-networkd

On nodes managed by systemd-networkd (e.g. Flatcar, Ubuntu Server), the routes and routing policy rules programmed by Submariner may be removed because systemd-networkd treats them as foreign. This requires systemd 249 or later (ManageForeignRoutes= was added in 246 and ManageForeignRoutingPolicyRules= in 249); on systemd 246–248 the routing policy rules can still be removed. On every gateway node, create /etc/systemd/networkd.conf.d/10-submariner-foreign-routes.conf with the following content and restart systemd-networkd:

[Network]
ManageForeignRoutes=no
ManageForeignRoutingPolicyRules=no

The drop-in prevents future removal; it does not restore entries that are already missing. To recover, restart the route agent on the affected gateway node so that it re-installs the rule and routes:

kubectl -n submariner-operator delete pod -l app=submariner-routeagent --field-selector spec.nodeName=<gateway node>

If the nodes are not reachable over SSH, use a privileged DaemonSet that does the work, for example:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: submariner-gateway-node-config
  namespace: submariner-operator
  labels:
    app: submariner-gateway-node-config
spec:
  selector:
    matchLabels:
      app: submariner-gateway-node-config
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: submariner-gateway-node-config
    spec:
      automountServiceAccountToken: false
      nodeSelector:
        submariner.io/gateway: 'true'
      tolerations:
        - operator: Exists
      terminationGracePeriodSeconds: 5
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: networkd-dropin
          image: docker.io/library/busybox:1.37.0
          command:
            - sh
            - -c
            - |
              set -eu
              f=/host/networkd.conf.d/10-submariner-foreign-routes.conf
              want='[Network]
              ManageForeignRoutes=no
              ManageForeignRoutingPolicyRules=no'
              if [ "$(cat "$f" 2>/dev/null)" = "$want" ]; then
                echo "$f up to date"
              else
                printf '%s\n' "$want" > "$f"
                chmod 0644 "$f"
                echo "$f written; effective at the next systemd-networkd start"
              fi
              trap 'exit 0' TERM INT
              while :; do sleep 3600 & wait $!; done              
          lifecycle:
            preStop:
              exec:
                command:
                  - sh
                  - -c
                  - rm -f /host/networkd.conf.d/10-submariner-foreign-routes.conf
          securityContext:
            runAsUser: 0
            runAsGroup: 0
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
          resources:
            requests:
              cpu: 1m
              memory: 8Mi
            limits:
              memory: 16Mi
          volumeMounts:
            - name: networkd-conf-d
              mountPath: /host/networkd.conf.d
      volumes:
        - name: networkd-conf-d
          hostPath:
            path: /etc/systemd/networkd.conf.d
            type: DirectoryOrCreate

Note that this is a privileged container and this might not be compatible with your security policies, adapt as needed.

When a node loses the submariner.io/gateway label, the DaemonSet pod stops running there and its preStop hook deletes the drop-in file. The change takes effect at the next systemd-networkd restart; restart it manually to apply the revert immediately. If the pod is terminated ungracefully, the hook does not run: delete /etc/systemd/networkd.conf.d/10-submariner-foreign-routes.conf on the former gateway node and restart systemd-networkd.

Support Matrix

Submariner is designed to be cloud provider agnostic, and should run in any standard Kubernetes cluster. Submariner has been tested with the following network (CNI) Plugins:

• OpenShift-SDN
• Weave
• Flannel
• Canal
• Calico (see the Calico-specific deployment instructions)
• OVN - Requires OVN NorthBound DB version 6.1.0+

Submariner supports all currently-supported Kubernetes versions, as determined by the Kubernetes release policy.

Deployment

Submariner is deployed and managed using its Operator. Submariner’s Operator can be deployed using subctl or Helm.

The recommended deployment method is subctl, as it is currently the default in CI and provides diagnostic features.