I did not set out to run a Kubernetes cluster at home. I thought my requirements were simple: I wanted to build a container for an application whose code was stored in my local GitLab instance, and deploy that container somewhere. I already had GitLab CE and the GitLab Runner set up on my local Proxmox VE instance.

The application wasn’t the problem, building and testing wasn’t the problem, but deploying updates automatically seemed a bit tricky. I’d deployed plenty of applications with Docker and Docker Compose before, but the more I dug into integrating that with a CI/CD workflow, the more ungainly it seemed. I’d initially been trying to avoid Kubernetes, but in the end I came around to the idea of deploying a Kubernetes cluster as part of my lab environment at home.

Once I’d come to terms with the idea that Kubernetes was the right way to do this, the next question was the actual mechanics of deploying from GitLab CI. Tempting as it may seem, just calling kubectl from GitLab CI is very much the wrong way to do this. The right way is Flux, but figuring out the whole arrangement from soup to nuts took me a ton of false starts.

Why Flux? Flux is what is known as a GitOps tool—the concept being that the state of your cluster should be defined by the contents of a Git repository, rather than whatever random YAML manifests someone might have applied with kubectl or the Kubernetes Dashboard. This allows you to easily version the state of the cluster and roll back undesirable changes, and it prevents there being “hidden state” which could get lost absent meticulous documentation.

So, complicated as it may be, this approach has its merits. Merely running kubectl apply at the end of a CI/CD pipeline gives you far less control and makes it harder to introspect the state of the cluster—sure, you might have a pile of YAML manifests on your computer which reflected the state of the application at the moment you first deployed it, but as your CI/CD pipeline deployed things on top of that, the cluster started to drift from the initial state. Now, without re-exporting those objects from the Kubernetes control plane, do you know what the current state is? And suppose you want to make some further modifications—how do you avoid trampling over the changes your CI/CD pipeline has made subsequently? The GitOps approach avoids these pitfalls by relying on manifests in a Git repository as the source of truth for the cluster’s state.

Here, I’ll try to outline briefly how I made this work with a minimum of overhead and fuss, and how I’ve achieved a setup that I’m actually quite happy with.

One of the things I find somewhat annoying about the Kubernetes ecosystem that many components have their own command-line administrative tools which have to be installed locally and kept up-to-date—there’s kubectl itself, cilium and hubble, flux, gitops for Weave GitOps, cmctl for cert-manager, and more. If you’re like me, you’ll just have to hold your nose and get over that. Fortunately, many of these are distributed with Homebrew, which eases the pain somewhat.

My first step was to spin up a new Debian 12 VM on my Proxmox server; you can do this in an LXC container, but it’ll become a headache, and especially if you have any thought of using Longhorn for storage you’ll want to be running in a proper VM or on bare metal rather than in an LXC container1.

I chose early on to deploy k3s rather than using kubeadm or other options; k3s is lighter-weight and easier to deploy than most of the alternatives. Still, I elected to disable k3s’s built-in Traefik Ingress controller and ServiceLB load balancer, as well as the Flannel CNI (the first and second because I would be deploying the Tailscale Kubernetes operator instead, and the third because I’d run into some adverse interactions between Flannel and systemd-networkd and so used Cilium instead).

Anyway, after installing k3s and Cilium I then installed the Tailscale Kubernetes operator, which made my cluster available from any machine on my tailnet without having to worry about certificates or authentication. The operator also makes it trivial to expose any Service or Ingress resource on a tailnet, a feature which I find immensely useful.

I then followed the GitLab tutorial to set up Flux and agentk, the GitLab Agent for Kubernetes. (It turns out that this will all work just fine without the GitLab Agent for Kubernetes, so you can omit that. The only downside to this is you’ll have to either manually configure webhooks or just wait for Flux to discover changes to the Git repositories it’s watching.)

I also deployed Weave GitOps per the documentation and the Kubernetes Dashboard (using oauth2-proxy and kube-oidc-proxy for authentication, which I’ll have to write about some other time), but these aren’t critical, and you can monitor just about everything using kubectl or k9s and the Flux CLI.

What were we trying to do, anyway? Oh, right, deploy a containerized application!

In my GitLab instance, I had two projects for this application: one containing the application code itself, and one containing the YAML manifests for the Kubernetes deployment (a Deployment, Service, and Ingress). I had already configured GitLab CI to build the application and push a container image to the GitLab container registry. From here, things actually started to get progressively easier. The first step was simply to configure a Flux GitRepository and Kustomization pointing to the deployment repository, which got those YAML manifests deployed on the cluster.

But what about updates? Suppose I pushed some new code, how does that get deployed? There’s a small prerequisite here, which is that your CI/CD job must be pushing container images tagged with an image tag that can be sorted to find the latest image. With that out of the way, there are three more Flux objects we have to bring into play to accomplish this, part of Flux’s image update automation:

  1. ImageRepository: to define the container registry which will be searched for images
  2. ImagePolicy: to define how Flux will find the latest image tag among those available in the registry
  3. ImageUpdateAutomation: to define how Flux will update manifests when a new version is found

Somewhat shockingly, this all just works! I can push new code to the application’s repository; GitLab CI will build and test, and if the tests pass the container image gets pushed to the GitLab Container Registry. Every ten minutes, the Flux ImageRepository refreshes itself, and then the ImagePolicy uses the configured regular expression and sorting rules to find the latest image. If there’s a new “latest image”, the ImageUpdateAutomation then commits an update to the deployment repository with the new container version, and then this update gets picked up by the Flux GitRepository and deployed by the Kustomize controller, exactly as if I had edited one of the YAML manifests myself. This, then, is the key to the whole process. While everything is automated, the entire process is transparent, and proceeds as if a human were in the loop. There’s no “hidden state”, and it’s easy to see precisely what got deployed and when, as well as to roll back to any arbitrary prior state.

The really smart thing behind the Flux image automation approach is that it would work all the same if there were multiple container images I wanted to watch for updates, if those images were in various different repositories, and so on. As long as they have automation-friendly tags which can be used to automatically determine the latest version, you can then use Flux to update YAML manifests, which can in turn be picked up and deployed by Flux.

You can even configure Flux to commit to a new branch (so that you could then open a pull request or merge request, for example), rather than directly committing to your repository’s main branch.

After all this, you might be feeling like the fox in the “I just want to serve five terabytes” video. I won’t (and can’t) deny that this is all fairly complicated. There are absolutely more brute-force ways to do this. Heck, you could do all of this with standalone Docker and no Kubernetes at all (for example, with Watchtower, which I knew about before, forgot, and then rediscovered while writing this post)! And while this may all feel like overkill to deploy a single application to a single-node Kubernetes cluster, the advantage is that it lays the groundwork for a path forward which is nearly infinitely scalable—in terms of nodes, applications, teams/developers, and so on. Is it too much for your hobby application at home? Only you can know. Personally, I think it’s really neat that I can commit an update, and a few minutes later it will be deployed and accessible via my tailnet without any further interaction on my part.

For me, this has been a worthwhile (if at times frustrating) learning experience. And frankly, I’ve greatly enjoyed the sense of accomplishment I’ve gotten from overcoming challenges while working on this project.

What are the key takeaways?

  • This stuff is seemingly infinitely complex, but can be broken down into tiny chunks which can be digested and understood. Take it in steps, and it will all come together eventually.
  • Use k3s; it’s far and away the best Kubernetes distribution for getting started quickly and has the advantage of being lightweight.
  • Tailscale really helps. Tailscale makes everything about running a homelab (or any network, for that matter) better, and the Kubernetes operator alleviates a range of authentication and networking annoyances.
  • Don’t let anyone tell you that you must have three nodes to run Kubernetes; single-node clusters are perfectly valid and will work just fine. They won’t be redundant, of course, but redundancy may not be a requirement for your use case! If you want to play around with multiple nodes for redundancy, but you aren’t actually interested in scalability, you can also deploy a bunch of small nodes as VMs just to see how the cluster responds when you (for example) unceremoniously terminate one of the worker nodes.
  • Don’t worry about storage, especially if you’re only deploying on a single node. The local path provisioner that k3s comes with is fine for most cases, and you can always deploy Longhorn if you need something more complex.
  • I found it very helpful to have a local GitLab instance deployed, but you could use Forgejo, GitHub, or the public GitLab.com service. The services which are most relevant here are Git repository hosting (to hold the codebase of the application being deployed), a container registry (to hold container images for the application once they’re built), CI/CD pipelines (to build and test the application), and an OIDC IdP (not for Kubernetes itself, which we can authenticate to using the Tailscale operator’s auth proxy, but rather other services, like Weave GitOps, the Kubernetes Dashboard, and others for which OIDC is the easiest way to set up authentication).

  1. This is due to an issue with how the kernel handles iSCSI in containers (or rather, fails to do so). ↩