The Gateway Engine component is deployed in each participating cluster and is responsible for establishing secure tunnels to other clusters.
The Gateway Engine has a pluggable architecture for the cable engine component that maintains the tunnels. The following implementations are available:
The cable driver can be specified via the
--cable-driver flag while joining a cluster using
subctl. For more information, please refer
WireGuard needs to be installed on Gateway nodes. See the WireGuard installation instructions.
Instances of the Gateway Engine run on specifically designated nodes in a cluster of which there may be more than one for fault tolerance. Submariner supports active/passive High Availability for the Gateway Engine component, which means that there is only one active Gateway Engine instance at a time in a cluster. They perform a leader election process to determine the active instance and the others await in standby mode ready to take over should the active instance fail.
The Gateway Engine is deployed as a DaemonSet that is configured to only run on nodes labelled with “submariner.io/gateway=true”.
The active Gateway Engine communicates with the central Broker to advertise
Cluster resources to the other clusters connected to the
Broker, also ensuring that it is the sole
Endpoint for its cluster. The
Route Agent Pods running in the cluster learn about the
Endpoint and setup the necessary infrastructure to route cross-cluster
traffic from all nodes to the active Gateway Engine node. The active Gateway Engine
also establishes a watch on the Broker to learn about the active
Cluster resources advertised by the other clusters. Once two clusters are
aware of each other’s
Endpoints, they can establish a secure tunnel through
which traffic can be routed.
If the active Gateway Engine fails, another Gateway Engine on one of the
other designated nodes will gain leadership and perform reconciliation to
Endpoint and to ensure that it is the sole
remote clusters will learn of the new
Endpoint via the Broker and establish
a new tunnel. Similarly, the Route Agent Pods running in the local cluster
automatically update the route tables on each node to point to the new active
Gateway node in the cluster.
The impact on datapath for various scenarios in a kind setup are captured in the following spreadsheet.
The Gateway Engine continuously monitors the health of connected clusters.
It periodically pings each cluster and collects statistics including basic connectivity,
round trip time (RTT) and average latency. This information is updated in the
resource. Whenever the Gateway Engine detects that a ping to a particular cluster has failed,
its connection status is marked with an error state. Service Discovery uses this information
to avoid unhealthy clusters during Service discovery.
The health checking feature can be enabled/disabled via an option on the
subctl join command.
The load balancer mode is still experimental, and is yet to be tested in all cloud providers nor in different failover scenarios.
The load balancer mode is designed to simplify the deployment of Submariner in cloud environments where worker nodes don’t have a dedicated public IP available.
When enabled for a cluster during
the operator will create a LoadBalancer type Service exposing both the encapsulation dataplane port
as well as the NAT-T discovery port. This load balancer targets Pods labeled with
When the LoadBalancer mode is enabled, the
preferred-server mode is enabled automatically for
the cluster, as IPsec is incompatible with the bi-directional connection mode and the
load balancers and client/server connectivity is necessary.
If a failover occurred, the load balancer would update to the new available and active gateway endpoints.
This mode is specific to the libreswan cable-driver which is based on IPsec. Other cable drivers ignore this setting.
When enabled for a cluster during
the gateway will try to establish connection with other clusters by configuring the IPsec connection
in server mode, and waiting for remote connections.
Remote clusters will identify the
preferred-server mode of this cluster, and attempt the connection.
This is useful in environments where on-premises clusters don’t have access to port mapping.
When both sides of a connection are in
preferred-server mode, they will compare the endpoint cable
names to decide which one will be server and which one will be client. When cable names are ordered
alphabetically, the first one will be the client, the second one will be the server.