Learn Kubernetes with k3s - part2 - Networking

This post is the continuation of part 1.

Let us look closely at the commands that we have used to install k3s.

master1.sudo( "curl -sfL https://get.k3s.io | sh -s - server "
              "--write-kubeconfig-mode 644 "
             f"--node-external-ip={server_ip} "
              "--flannel-external-ip "
              "--flannel-backend=wireguard-native "
             f"--tls-san {server_external_hostname}")

The first flag, --write-kubeconfig-mode 644 tells the process to produce a kubeconfig file in 644 mode, so that we can read it, copy it to our workstation to make the first connection to the cluster.

A kubeconfig file is used to configure access to Kubernetes clusters. It contains information such as authentication details, clusters, users and namespaces.

Next,

f"--node-external-ip={server_ip} "
 "--flannel-external-ip "
 "--flannel-backend=wireguard-native "

These flags set the node external IP to the server's static ip (we've set that to 192.168.1.53) and configure the networking plugin (flannel) to use that for routing.

There is a corresponding flag in the worker node command:

f"--node-external-ip={cxn.host}"

To test whether the cluster networking is working fine. Run the following commands:

$ curl -k https://192.168.1.53
404 page not found

$ curl -k https://192.168.1.54
404 page not found

You should see a 404 response for each.

This is because there is a traefik reverse-proxy service running at port 80 and 443 on every node, installed during cluster installation. Yet we have not defined any routing rules, hence the 404 errors.

Run this command to see all services running on the cluster:

$ kubectl get service -A
NAMESPACE     NAME             TYPE           CLUSTER-IP    EXTERNAL-IP                              PORT(S)                      AGE
default       kubernetes       ClusterIP      10.43.0.1     <none>                                   443/TCP                      71m
kube-system   kube-dns         ClusterIP      10.43.0.10    <none>                                   53/UDP,53/TCP,9153/TCP       71m
kube-system   metrics-server   ClusterIP      10.43.22.2    <none>                                   443/TCP                      71m
kube-system   traefik          LoadBalancer   10.43.158.9   192.168.1.53,192.168.1.54                80:30824/TCP,443:30872/TCP   71m

Traefik is in the last row of the table.

If you did not get a reply for any curl command, it means the networking has failed. This was the case when I used the default installation command from the official quick-start guide.

I had to reinstall k3s with the above-mentioned flags to make networking works.

Note the special flag--flannel-backend=wireguard-native

Use WireGuard to encapsulate and encrypt network traffic. May require additional kernel modules.

Sometimes the default installation command is not able to figure out which networking interface should be used for traffic routing. We remind it using the explicit flags.

For your curiosity, here are all the different options for flannel.

To fiddle with these, we need a way to uninstall k3s from all nodes and reinstall k3s. Add the following fabric tasks:

@task
def uninstall_agents(c):
    ThreadingGroup(*all_workers).sudo("/usr/local/bin/k3s-agent-uninstall.sh")


@task
def uninstall_server(c):
    Connection("k3s-master1").sudo("/usr/local/bin/k3s-uninstall.sh")

$ fab uninstall-agents
$ fab uninstall-server

You can then tap on these to test which networking options work for your setup.

Read this in-depth article "Deciphering the Kubernetes Networking Maze: Navigating Load-Balance, BGP, IPVS and Beyond" to understand more about Kubernetes networking.

The next flag, --tls-san {server_external_hostname} tells the process to generate a TLS certificate with server_external_hostname (example.com) as SAN, so that we can access the cluster through the external hostname for convenience.

Next,

master1.get("/etc/rancher/k3s/k3s.yaml", local="kubeconfig")

subprocess.run([
    "sed", "-i", f"s/127.0.0.1/{server_external_hostname}/g", "kubeconfig"
])
subprocess.run(["chmod", "600", "kubeconfig"])

These lines copy kubeconfig to the workstation with the external hostname and file mode.

And finally the installation command at the worker node,

node_token = master1.sudo(
    "cat /var/lib/rancher/k3s/server/node-token", 
    hide=True
).stdout.strip()
for cxn in ThreadingGroup(*all_workers):        
    cxn.sudo( "curl -sfL https://get.k3s.io | "
             f"K3S_URL=https://{server_ip}:6443 "
             f"K3S_TOKEN={node_token} "
......

We recover the node token from the server node, and use it to configure the worker node, as instructed in the official quick-start guide.

To be continued in part 3, about kubectl.