Learn Kubernetes with k3s - part1 - Installation

In 2024, as far as learning Kubernetes is concerned we have a myriad of tools at our disposal:

1. mini distributions such as minikube, kind, k0s, k3s, etc.
2. Q&A AI: ChatGPT, Gemini, Mistral, etc.

It makes me wonder if one can learn Kubernetes administration and application development in under 1 week. I did it in 3 months many years ago.

There is a systematic approach I have adopted successfully for many learning tasks:

Hack a mini project and ask AI models the right questions.

I believe when doing it right, it can help you quickly grasp Kubernetes' core concepts, setting you on a path to becoming a proficient user.

Let's get started with hacking with k3s.

K3s

K3s is a lightweight Kubernetes distribution that aims to simplify deploying and managing Kubernetes clusters. It is developed by Rancher Labs and is certified by the Cloud Native Computing Foundation (CNCF) as being fully compatible with Kubernetes.

K3s packages Kubernetes into a single binary file that is only around 45MB in size, much smaller than a full Kubernetes installation. It is designed for running Kubernetes on low-resource environments like edge/IoT devices.

Even though it is lightweight, k3s maintains full Kubernetes functionality and is interoperable with tools that work with regular Kubernetes, making it an excellent choice for hands-on learning and experimentation.

Getting the Servers

There are many options, I leave it to you to decide.

1. Get instances from a cloud provider
2. Set up Rasberry PIs at home
3. Install guest OS' with Vagrant / Virtualbox
4. Or (if you happen to use a Linux desktop) use your OS directly

I chose 3 and installed two guests on two home PCs separately, one as the server, another as the agent (worker node). Both host machines are on the same network.

The Vagrantfile I used looks like this:

Vagrant.configure("2") do |config|
  config.vm.hostname = "k3s-master1"
  config.vm.box = "generic/ubuntu2204"
  config.vm.network "public_network", ip: "192.168.1.53", bridge: "en0: Ethernet"

  config.vm.provider "virtualbox" do |v|
    v.memory = 8000
    v.cpus = 2
  end
end

I omit the Vagrantfile file for the worker node as it is only slightly different. Note that I have configured static IPs for both nodes.

To spin the guest nodes up, run this command on both hosts

$ vagrant up

Install K3s

At this point, I have two vagrant nodes up. To install k3s, I use an automation tool Fabric to write the necessary installation tasks in a fabfile.py.

First some settings:

SSH config:

$ cat ~/.ssh/config 
Host k3s-master1
	User vagrant
	HostName 192.168.1.53

Host k3s-worker1
	User vagrant
	HostName 192.168.1.54

fabfile.py opening:

import subprocess

from fabric import ThreadingGroup, Connection, task
from fabric.exceptions import GroupException

ips = {
    "k3s-master1": "192.168.1.53",
    "k3s-worker1": "192.168.1.54",
}
server_ip = ips["k3s-master1"]
server_external_hostname = "example.com"
all_workers = [host for host in ips if "worker" in host]

example.com is a domain I registered for the k3s API server so that I can use it to access my cluster over the internet. I have set a DNS A record pointing it to the public IP address of my router.

Then the task to upgrade the OS on both nodes (optional)

@task
def upgrade_os(c):
    all_hosts = ThreadingGroup(*ips.keys())
    all_hosts.sudo("apt update -y")
    all_hosts.sudo("apt upgrade -y")

    try:
        all_hosts.sudo("reboot")
    except GroupException:
        pass

Run this command to execute the task

$ fab upgrade-os

Then the tasks to install k3s, first on the server node, then on the worker node

@task
def install_server(c):
    master1 = Connection("k3s-master1")

    master1.sudo( "curl -sfL https://get.k3s.io | sh -s - server "
                  "--write-kubeconfig-mode 644 "
                 f"--node-external-ip={server_ip} "
                  "--flannel-external-ip "
                  "--flannel-backend=wireguard-native "
                 f"--tls-san {server_external_hostname}")

    master1.get("/etc/rancher/k3s/k3s.yaml", local="kubeconfig")

    subprocess.run([
        "sed", "-i", f"s/127.0.0.1/{server_external_hostname}/g", "kubeconfig"
    ])
    subprocess.run(["chmod", "600", "kubeconfig"])


@task
def install_agents(c):
    master1 = Connection("k3s-master1")

    node_token = master1.sudo(
        "cat /var/lib/rancher/k3s/server/node-token", 
        hide=True
    ).stdout.strip()

    for cxn in ThreadingGroup(*all_workers):        
        cxn.sudo( "curl -sfL https://get.k3s.io | "
                 f"K3S_URL=https://{server_ip}:6443 "
                 f"K3S_TOKEN={node_token} "
                  "sh -s - agent "
                 f"--node-external-ip={cxn.host}")

Run these commands to execute the tasks

$ fab install-server
$ fab install-agents

Test the k3s Installation

$ mv kubeconfig ~/.kube/config

Note: the mv command will overwrite your existing config file.

Install your kubectl command, enable port forwarding from your home router to the server node at port 6443, then run

$ kubectl get nodes
NAME          STATUS   ROLES                  AGE   VERSION
k3s-master1   Ready    control-plane,master   67m   v1.28.7+k3s1
k3s-worker1   Ready    <none>                 63m   v1.28.7+k3s1

You should see both nodes in Ready state.

Check all pods are running fine

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS       AGE
kube-system   helm-install-traefik-8frcc                0/1     Completed   2              12h
kube-system   helm-install-traefik-crd-gbnjw            0/1     Completed   0              12h
kube-system   svclb-traefik-5d5abbc4-95jnq              2/2     Running     2 (10h ago)    12h
kube-system   traefik-f4564c4f4-hxws8                   1/1     Running     1 (10h ago)    12h
kube-system   coredns-6799fbcd5-zvmtd                   1/1     Running     1 (10h ago)    12h
kube-system   local-path-provisioner-6c86858495-gbnjx   1/1     Running     2 (10h ago)    12h
kube-system   metrics-server-78bcc4784b-gv7kp           1/1     Running     2 (10h ago)    12h
kube-system   svclb-traefik-5d5abbc4-sfww6              2/2     Running     6 (4m8s ago)   11h
kube-system   svclb-traefik-5d5abbc4-7bqn9              2/2     Running     0              7h59m

Try creating a pod

$ kubectl run busybox --image=busybox --restart=Never -it -- uname -r
5.15.0-100-generic

We now have a working k3s cluster!

To be continued in part 2, networking.