My Homelab! Pt. 1

If you’re in the tech sphere, you might have heard the term Homelab (or homelabbing) before. If you haven’t, its basically just 1-n computers that you have at home that you can use as servers. There are a ton of interesting use cases for them. You can over complicate your life, annoy your spouse and children, and even use stop users on your network from using the internet! The joke here being that most homelabbers tend to alter their homes and networks in a way that makes it impossible for anyone but them to use or update it. I have used mine for HomeAssistant, Plex, personal and professional website (like the one you’re reading now), and so much more. Above all else though, I use mine to play with Kubernetes.

Kubernetes is one of my favourite pieces of software. It originated at Google where it lived (and I believe still lives) under the name Borg. It lets you connect a group of physical servers, like an old desktop, laptop, or raspberry pi into a ‘cluster’ and give them tasks! The cluster then determines how best to complete those tasks and which workers in the cluster are able to complete them. For instance if your task needs a GPU, you tell the cluster that and it will only schedule your task on the worker that has the GPU. It makes me feel like the commander of a vast fleet, whom with the stroke of my keyboard do my bidding. To the outside world though it just sounds like some clicks followed by an increase in fan volume as the 8 year old CPUs attempt to pull down an image I hastily slapped together. Still feels cool though.

Rise of the machines

My original half-lie to myself on why I was going to spend thousands of dollars on defunct enterprise equipment was so that I had a playground for kubernetes that would let me become a better DevOps / Sys Admin / Architect. I say half-lie because it really was just something I was interested in and wanted to do, but I needed something more concrete to calm the family as I stroked my new Dell R730xd LFF while whispering “my preciousssss”. They don’t knows the powers of preciouseses 48 cores-eses.

The early days of the homelab were less glamourous than my dell server. I started off with a desktop that was previously the best you could buy. Absolute top of the line. In 2009. Intel i7-960 (no I’m not missing a zero), 4gb of RAM, 256 GB of storage (RAID 10), a GTX 1070, and a 500W power supply that at one point released the magic smoke. I used to play so much minecraft on that badboy. When I built my now desktop I retired the old machine, named her Martha, and made her the first of the collective. If you’ve got a render job, or FFMPEG workload, or some (small) AI inference you want done, Martha is your gal.

I think it really is a testament to the machines that we can create as a species. I have been using that machine almost daily for the past decade and a half.

I started the homelab with Martha and two Raspberry Pis, a Pi3 and a Pi4. While it ran, it wasn’t really the work horse I was looking for. My calls to the control plane API, the thing that is minimum overhead for the cluster, were slow. That didn’t bode well for the actual workloads I wanted to run. I ended up pausing getting the cluster running and salvaged the Pi 4 for my 3D printer (running Octoprint), so I never saw the presumed horrors of a cluster with a Pi3. Martha was enough to run an NFS, PiHole, and some small cron jobs I cobbled together.

When I acquired the Heather (The artist formally known as Dell 730xd LFF), that’s when I decided a home cluster was in the cards. Heather runs proxmox, a virtualization software that lets you easily manage virtual machines and their configuration. This way I can turn Heather from one machine with lots of memory and cores, into many machines with fewer. My workers have anywhere from 4 to 8 cores, and 8-64 gb of RAM. Currently there are 6 workers (5 VMs + Martha) and 3 control plane nodes. You need a minimum of 3 nodes to run in HA (High Availability) mode, and as that is the mode you should be running in Production I wanted to replicate that locally. High availability means a few things in this context, but mostly it lets you be sure that should a node go down, other nodes will take their place. Unfortunately in my set up, I’ve kind of borked that feature. If Heather goes down, the entire control plane goes down and Martha is left on her lonesome unsure of what to do. This is trade off I’m aware of and okay with. The goal here is to learn and I’ve already got a lot of hardware gobbling up watts, I don’t want to add anymore for the moment.

> k get nodes
NAME            STATUS   ROLES                       AGE    VERSION
k3s-control-1   Ready    control-plane,etcd,master   664d   v1.30.6+k3s1
k3s-control-2   Ready    control-plane,etcd,master   664d   v1.30.6+k3s1
k3s-control-3   Ready    control-plane,etcd,master   664d   v1.30.6+k3s1
k3s-worker-1    Ready    <none>                      646d   v1.30.6+k3s1
k3s-worker-2    Ready    <none>                      664d   v1.30.6+k3s1
k3s-worker-3    Ready    <none>                      664d   v1.30.6+k3s1
k3s-worker-4    Ready    <none>                      646d   v1.30.6+k3s1
k3s-worker-5    Ready    <none>                      646d   v1.30.6+k3s1
martha          Ready    <none>                      118d   v1.30.6+k3s1

You can see these nodes are approaching their second birthday. They grow up so fast :’). Martha, while being actually the oldest, has the most recent date here as her 10 year old SSD gave up the ghost about 130 days ago, and I had to re-image a fresh drive and add her back to the cluster.

They’ve held up well for my use cases, and I’ve learned a lot. I wouldn’t be the swiss army knife of Kubernetes I am today without spending many an evening debugging my loadbalancers and network settings. That’s not to say there isn’t more to learn, or more to tell you about! My next post is all about the software that runs on the cluster. I’ll what I set up and how, maybe you’ll find something interesting there!