Google (GKE)

This quickstart guide covers the necessary steps to deploy two Google Kubernetes Engine (GKE) clusters on Google Cloud Platform (GCP) using gcloud. The clusters are named cluster-a and cluster-b.

Once the GKE clusters are deployed, we deploy Submariner with Service Discovery to interconnect the two clusters. Note that this guide focuses on Submariner deployment on clusters with non-overlapping Pod and Service CIDRs.

The guide assumes that you have a working GCP account and the gcloud binary installed and configured. To get the binary, please visit Google’s website.

Cluster Creation

Create two identical Kubernetes clusters on GKE. For this guide, the following minimal configuration was used, however not everything is required (see the note part below).

gcloud container clusters create "cluster-a" \
    --zone "europe-west3-a" \
    --cluster-ipv4-cidr ""
    --cluster-version "1.17.13-gke.1400" \
    --username "admin" \
    --machine-type "g1-small" \
    --image-type "UBUNTU" \
    --disk-type "pd-ssd" \
    --disk-size "15" \
    --num-nodes "3" \
    --network "default" \
gcloud container clusters create "cluster-b" \
    --zone "europe-west3-a" \
    --cluster-ipv4-cidr ""
    --cluster-version "1.17.13-gke.1400" \
    --username "admin" \
    --machine-type "g1-small" \
    --image-type "UBUNTU" \
    --disk-type "pd-ssd" \
    --disk-size "15" \
    --num-nodes "3" \
    --network "default" \

Make sure to use Kubernetes version 1.17 or higher, set by --cluster-version (refer to the GKE docs/ release notes). Also double check for non-overlapping Pod and Service CIDRs between clusters using the commands below:

gcloud container clusters describe "cluster-a" --zone "europe-west3-a" | grep -e clusterIpv4Cidr -e servicesIpv4Cidr
gcloud container clusters describe "cluster-b" --zone "europe-west3-a" | grep -e clusterIpv4Cidr -e servicesIpv4Cidr

Prepare Clusters for Submariner

Next, the created clusters need some changes in order for Submariner to successfully open the IPSEC tunnel between them.

Preparation: Node Configuration

As of version 0.7 of Submariner (the current one while writing this), Google’s native CNI plugin is not directly supported. GKE clusters can be generated with Calico CNI instead, but this was not tested during this demo and therefore could hold surprises as well.

So as this guide uses Google’s native CNI plugin, configuration is needed for the eth0 interface of each node on every cluster. The used workaround deploys netshoot pods onto each node that configure the reverse path filtering. The scripts in this Github repository need to be executed in all clusters.

chmod +x
chmod +x
kubectx cluster-a # kubectl config use-context cluster-a
kubectx cluster-b # kubectl config use-context cluster-b

Preparation: Firewall Configuration

Submariner requires UDP ports 500, 4500 and 4800 to be open in both directions. Additionally the microservices’ traffic needs to flow through the IPsec tunnel as TCP packets. Hence the TCP traffic has source and destination addresses originating in the participating clusters. Create those firewall rules on the GCP project.

Use the commands above to retrieve the clusterIpv4Cidr and servicesIpv4Cidr for both clusters. If you copied the above code, you can just copy&paste the values below for the --source-ranges and --destination-ranges.

gcloud compute firewall-rules create "allow-tcp-in" --allow=tcp \
  --direction=IN --source-ranges=,,,

gcloud compute firewall-rules create "allow-tcp-out" --allow=tcp --direction=OUT \

gcloud compute firewall-rules create "udp-in-500" --allow=udp:500 --direction=IN
gcloud compute firewall-rules create "udp-in-4500" --allow=udp:4500 --direction=IN
gcloud compute firewall-rules create "udp-in-4800" --allow=udp:4800 --direction=IN

gcloud compute firewall-rules create "udp-out-500" --allow=udp:500 --direction=OUT
gcloud compute firewall-rules create "udp-out-4500" --allow=udp:4500 --direction=OUT
gcloud compute firewall-rules create "udp-out-4800" --allow=udp:4800 --direction=OUT

After this, the clusters are finally ready for Submariner!

Deploy Submariner

Download the subctl binary and make it available on your PATH.

curl -Ls | bash
export PATH=$PATH:~/.local/bin
echo export PATH=\$PATH:~/.local/bin >> ~/.profile

We will deploy the Broker on cluster-a. The command will output a file named broker-info.subm to the directory it is run from, which will be used to setup the IPsec tunnel between clusters.

The Broker cluster’s API must be accessible by all the participating clusters, which is given by GKE standard configuration.

Kubecontext and the kubeconfig file can be specified (it defaults to ~/.kube/config):

subctl deploy-broker --kubecontext cluster-a --kubeconfig <path/to/config>

Verify the Broker components are installed using kubectl:

kubectl get crds --context cluster-a | grep -i submariner
kubectl get ns --context cluster-a | grep -i submariner
  > submariner-k8s-broker

Now it is time to register every cluster in the future clusterset to the Broker.

First join the Broker cluster itself (the Broker could also be deployed on a cluster that does not run workloads, hence installing the Broker does not automatically add the cluster). The command tags the cluster for Submariner with the ID cluster-a.

subctl join --kubecontext cluster-a --kubeconfig <path/to/config> broker-info.subm --clusterid cluster-a

Submariner will figure out most information needed on its own. In addition, you will see a dialogue on the terminal that asks you to

  • decide which of the three nodes shall be the Gateway. This node gets annotated with true automatically.
  • input the cluster’s Pod and Service CIDR, if it could not be detected.

When a cluster is joined, the Submariner Operator is installed. It creates several components in the submariner-operator namespace:

  • a submariner-gateway DaemonSet, to open a gateway for the IPsec tunnel on one node
  • a submariner-routeagent DaemonSet, which runs on every worker node in order to route the internal traffic to the local gateway via VXLAN tunnels
  • a submariner-lighthouse-agent Deployment, which accesses the Kubernetes API server in the Broker cluster to exchange Service information with the Broker
  • a submariner-lighthouse-coredns Deployment, which - as an external DNS server - gets forwarded requests to the *.clusterset.local domain for cross-cluster communication by Kubernetes’ internal DNS server

Check the DaemonSets and Deployments with the following command:

kubectl --context cluster-a get ds,deploy -n submariner-operator

Repeat with the second cluster:

subctl join --kubecontext cluster-b --kubeconfig <path/to/config> broker-info.subm --clusterid cluster-b

Then verify connectivity and CIDR settings within the clusterset using:

subctl show all

You should see similar output to this:

cluster-a     35.x.x.x    strongswan      local
cluster-b     35.x.x.x    strongswan      remote

gke-*-*    cluster-a    strongswan,    connected

gke-*-*    active        All connections (1) are established

Final Workaround for KubeDNS

GKE uses KubeDNS by default for cluster-internal DNS queries. Submariner however only works with CoreDNS as of version 0.7. As a consequence, the *.clusterset.local domain stub needs to be added manually to KubeDNS. Query the ClusterIP of the submariner-lighthouse-coredns Service in cluster-a:

CLUSTER_IP=$(kubectl --context cluster-a get svc submariner-lighthouse-coredns -n submariner-operator \
  -o=custom-columns=ClusterIP:.spec.clusterIP | tail -n +2)

Use the information to apply the following ConfigMap, automatically replacing $CLUSTER_IP with the ClusterIP obtained for the submariner-lighthouse-coredns:

cat <<EOF | kubectl --context cluster-a apply -f -
apiVersion: v1
kind: ConfigMap
  stubDomains: |
  labels: EnsureExists
  name: kube-dns
  namespace: kube-system

Repeat both commands, the CLUSTER_IP discovery and the kubectl apply, this time using --context cluster-b.

Verify Deployment

To manually verify the deployment, follow the steps below using either a headless or ClusterIP nginx service deployed in cluster-b.

Deploy ClusterIP Service
export KUBECONFIG=cluster-b/auth/kubeconfig
kubectl -n default create deployment nginx --image=nginxinc/nginx-unprivileged:stable-alpine
kubectl -n default expose deployment nginx --port=8080
subctl export service --namespace default nginx
Deploy Headless Service

Note that headless Services can only be exported on non-globalnet deployments.

export KUBECONFIG=cluster-b/auth/kubeconfig
kubectl -n default create deployment nginx --image=nginxinc/nginx-unprivileged:stable-alpine
kubectl -n default expose deployment nginx --port=8080 --cluster-ip=None
subctl export service --namespace default nginx

Run nettest from cluster-a to access the nginx service:

export KUBECONFIG=cluster-a/auth/kubeconfig
kubectl -n default  run --generator=run-pod/v1 tmp-shell --rm -i --tty --image -- /bin/bash
curl nginx.default.svc.clusterset.local:8080

To access a Service in a specific cluster, prefix the query with <cluster-id> as follows:

curl cluster-a.nginx.default.svc.clusterset.local:8080

Verify StatefulSets

A StatefulSet uses a headless Service. Create a web.yaml as follows:

apiVersion: v1
kind: Service
  name: nginx-ss
  labels: nginx-ss nginx-ss
  - port: 80
    name: web
  clusterIP: None
  selector: nginx-ss nginx-ss
apiVersion: apps/v1
kind: StatefulSet
  name: web
  serviceName: "nginx-ss"
  replicas: 2
    matchLabels: nginx-ss nginx-ss
      labels: nginx-ss nginx-ss
      - name: nginx-ss
        image: nginxinc/nginx-unprivileged:stable-alpine
        - containerPort: 80
          name: web

Use this yaml to create a StatefulSet web with nginx-ss as the Headless Service.

export KUBECONFIG=cluster-a/auth/kubeconfig
kubectl -n default apply -f web.yaml
subctl export service -n default nginx-ss
curl nginx-ss.default.svc.clusterset.local:8080

To access the Service in a specific cluster, prefix the query with <cluster-id>:

curl cluster-a.nginx-ss.default.svc.clusterset.local:8080

To access an individual pod in a specific cluster, prefix the query with <pod-hostname>.<cluster-id>:

curl web-0.cluster-a.nginx-ss.default.svc.clusterset.local:8080

Perform automated verification

This will perform automated verifications between the clusters.

subctl verify cluster-a/auth/kubeconfig cluster-b/auth/kubeconfig --only service-discovery,connectivity --verbose

Reconfig after Node Restart

If the GKE Nodes were at some point drained or deleted, the Submariner Pods needed to terminate. Once the Nodes are up again, remember to

  • label one Node with kubectl label node <name> in order for the Gateway to be deployed on this Node
  • apply the Node Configuration workaround again
  • change the applied KubeDNS workaround to reflect the current submariner-lighthouse-coredns IP.

This makes Submariner functional again and work can be continued.