Raspberry Pi K3S Cluster - Part 1

Posted on Oct 30, 2024

Earlier this month I decided to pick up a few Raspberry Pi devices to create small, lower powered bare metal Kubernetes cluster. With the rising costs of electricity, and to keep my home cooler in the summer, I wanted an option where I can run applications with redundancy year round without a significant power bill.

Due to the expanse of this project I will break it up into several parts. This part will only cover setting up the nodes. It will not go into installation and configuration of any networking or storage. Software such as monitoring and security will have their own posts.

Prerequisits

At the very least you need three nodes, one control plane and two worker nodes. For each node you need a USB-C cable for power and a micro-SD card with some sort of storage for the workers for persistent volumes. This can be a flash drive, SSD, or NVMe. Below is what I purchased and will be using for this tutorial.

Hardware

Software

  • Operating System: Raspberry Pi OS Lite
  • Kubernetes: K3S – Lightweight Kubernetes
  • Utilities:
    • helm - Kubernetes package manager
    • k9s - Terminal UI for managing Kubernetes Cluster

Setup

The following is an outline of how the cluster will be setup:

  1. Burn Rapsberry Pi image to microSD cards. Make sure that you enable ssh and set the hostname. Then install the microSD card into the Raspberry PI
  2. Prepare all the nodes: load necessary kernel drivers, update OS, install necessary packages, etc.
  3. Partition and format the USB drives in the nodes.
  4. Install K3S on control plane.
  5. Install K9S and Helm on the control plane.
  6. Install K3S agent on nodes.

Burn the Raspberry Pi image and Prepare Each Node

Use the Raspberry Pi Imager to write Raperberry Pi OS Lite (64-bit) to the microSD card, which is found under Raspberry Pi OS (other) in the Raspberry Pi OS List. Before actually writing to the card make sure that you click on the Edit Settings button.

At the very least set the hostname, username and password, and locale settings.

I highly recommend adding your public key here.

Once the image has been burned to the microSD card, boot the Pi, and check to make sure that there is a minimal amount of RAM dedicated to the GPU since we will be running these as headless servers. It should be set to 4M.



sudo vcgencmd get_mem gpu
gpu=4M

If not then edit /boot/firmware/config.txt and add gpu_mem=4M to the bottom of the file.

Configures All Nodes

The following script will configure the control plane(s) and workers so they meet the requirements of K3S.



modprobe br_netfilter
cat < /etc/modules-load.d/k8s.conf
br_netfilter
EOF 
cat < /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
systemctl disable --now dphys-swapfile
apt update && apt upgrade -y && apt -y full-upgrade
apt install -y libraspberrypi-bin open-iscsi vim tmux
cp /boot/firmware/cmdline.txt ~
sed -i 's/$/ cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory/' /boot/firmware/cmdline.txt 
systemctl reboot

Bootstrap the Main Control Plane Node

The first step is to create a token and run the command to download and install K3S.



K3S_TOKEN=$(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32; echo)
curl -sfL https://get.k3s.io | sh -s - --flannel-backend none --token $K3S_TOKEN --disable-network-policy --disable-kube-proxy --disable servicelb --disable 'traefik' --disable 'local-storage'  --cluster-cidr=10.42.1.0/24 --service-cidr=10.43.1.0/24
[INFO]  Finding release for channel stable
[INFO]  Using v1.32.5+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.32.5+k3s1/sha256sum-arm64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.32.5+k3s1/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  Host iptables-save/iptables-restore tools not found
[INFO]  Host ip6tables-save/ip6tables-restore tools not found
[INFO]  systemd: Starting k3s
eche $K3S_TOKEN
k3s-token

Note: If you get an error about systemd not being able to start the service, run the curl command again.

Since we’re going to be using Cilium, we need to set --flannel-backend to none as well as disable network policy with --disable-network-policy. We want to disable servicelb and traefik since we will use Cilium for them. local-storage is also disabled because we will be using Longhorn.

Next, download and install Helm and K9S. When installing each of these, update the version variable to the latest release.



VERSION="v3.18.2"
curl -o - https://get.helm.sh/helm-$VERSION-linux-arm64.tar.gz | tar zxvf - 
chmod +x linux-arm64/helm 
mv linux-arm64/helm /usr/local/sbin/ 
rm -rf linux-arm64

For K9S



VERSION="v0.50.6"
curl -L -O https://github.com/derailed/k9s/releases/download/$VERSION/k9s_linux_arm64.deb && dpkg -i k9s_linux_arm64.deb && rm k9s_linux_arm64.deb

Add Worker Nodes

Retrieve the token that was generated on the control plane node and pass it along in the agent install command. Do this on all worker nodes and wait for the CNI to deploy the necessary pods on the workers.



K3S_TOKEN="k3s-token"
curl -sfL https://get.k3s.io | sh -s - agent --server https://k3s-cp-01:6443 --token $K3S_TOKEN
[INFO]  Finding release for channel stable
[INFO]  Using v1.32.5+k3s1 as release
[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.32.5+k3s1/sha256sum-arm64.txt
[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.32.5+k3s1/k3s-arm64
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Skipping installation of SELinux RPM
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO]  systemd: Enabling k3s-agent unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s-agent.service → /etc/systemd/system/k3s-agent.service.
[INFO]  Host iptables-save/iptables-restore tools not found
[INFO]  Host ip6tables-save/ip6tables-restore tools not found
[INFO]  systemd: Starting k3s-agent

Each of the worker nodes will have an additional storage device attached to it for persistent storage in the cluster. We need to format it so it can be used.



parted /dev/sda -- mklabel gpt mkpart
mkfs.ext4 /dev/sda1
mkdir /srv/storage
mount /dev/sda1 /srv/storage

Control Plane



kubectl label nodes k3s-node-01 kubernetes.io/role=worker
kubectl label nodes k3s-node-02 kubernetes.io/role=worker
kubectl label nodes k3s-node-03 kubernetes.io/role=worker

Conclusion

Technically you have a cluster now. However, it’s not fully functional. The next step is to install the container network interface and that is detailed in the next part.