Submariner is a tool built to connect overlay networks of different Kubernetes clusters. These clusters can be on different public clouds or on-premise. An important use case for Submariner is to connect disparate independent clusters into a single cohesive multi-cluster fleet.
However, by default, a limitation of submariner is that it doesn’t handle overlapping CIDRs (ServiceCIDR and ClusterCIDR) across these clusters. Each cluster must use distinct CIDRs that don’t conflict or overlap with any other cluster that is going to be part of cluster fleet.
This is largely problematic because most actual deployments use the default CIDRs for a cluster so every cluster ends up using the same CIDRs. Changing CIDRs on existing clusters is a very disruptive process and requires a cluster restart. So submariner needs a way to allow clusters with overlapping CIDRs to connect together.
To support overlapping CIDRs in clusters connected through submariner, submariner has a component called Global Private Network, GlobalNet (
globalnet). This GlobalNet is a virtual network specifically to support submariner’s multi-cluster solution with a global CIDR. Each cluster is given a subnet from this virtual Global Private Network, configured as new cluster parameter
GlobalCIDR (e.g. 169.254.0.0/16) which is configurable at time of deployment.
User can also manually specify GlobalCIDR for each cluster that is joined to the broker using the flag
globalnet-cidr passed to
subctl join command. If Globalnet is not enabled in the broker or if a GlobalCIDR is preconfigured in the cluster, the supplied globalnet-cidr will be ignored.
Once configured, each Service and Pod that requires cross-cluster access is allocated an IP, named
globalIp, from this
GlobalCIDR that is annotated on the Pod/Service object. This globalIp is used for all cross-cluster communication to and from a Pod and the globalIp of a remote Service. Routing and IPTable rules are configured to use the globalIp for ingress and egress. All address translations occur on the Gateway node.
Unlike vanilla Submariner, where Pod to Pod connectivity is also supported, GlobalNet only supports Pod to remote Service connectivity using globalIps.
Submariner GlobalNet is a component that provides cross-cluster connectivity from Pods to remote Services using their globalIps. Compiled as binary
submariner-globalnet, it is responsible for maintaining a pool of global IPs, allocating IPs from the GlobalIp pool to pods and services, annotating Services and Pods with their globalIp, and configuring the required rules on the gateway node to provide cross-cluster connectivity using globalIps.
Globalnet also supports connectivity from the Nodes (including Pods that use HostNetworking) to globalIp of remote Services.
It mainly consists of two key components: the IP Address Manager and Globalnet.
The IP Address Manager (IPAM) component does the following:
GlobalCIDRconfigured on cluster.
This component is responsible for programming the routing entries, IPTable rules and does the following:
kube-proxyIPTables chain which in turn directs traffic to Service’s backend Pods.
Globalnet currently relies on
kube-proxy and thus will only work with deployments that use
Connectivity is only part of the solution as pods still need to know the IPs of services on remote clusters.
This is achieved by enhancing lighthouse with support for GlobalNet. The Lighthouse controller adds the service’s globalIp to the
ServiceImport object that is distributed to all clusters. The lighthouse plugin then uses the Service’s globalIp when replying to DNS queries for the Service.
Nothing extra needs to be done to build
submariner-globalnet as it is built with the standard submariner build.