UPDATE 17.02.2022: k3d moved to a new GitHub organization @ k3d-io/k3d
k3d is my first large open-source project and also my first big project written in Go. In fact, it’s my first big (and serious) programming project ever.
Everytime I’m talking somewhere, participating in a meetup or presenting something even only remotely related to k3d (ok, honestly, that’s the only thing most people know me for), people ask me, how I came to write k3d… so I decided to write it down, so I have to talk less in the future 😄
As most projects, k3d also started with a problem:
In 2018, I started working as a DevOps-Engineer in a rather small backend team at trivago.
The team was running some pretty complex microservices on an on-premise bare-metal cluster (no virtual machines involved) hosted in one of trivago’s data centers.
Early on they had adopted containers and thus were developing their Python applications locally with a Docker and docker-compose based toolchain.
The deployed containers were managed by Rancher 1.6 running on top of Docker on the bare-metal Debian servers.
Rancher 1.6 provided a docker-compose based container orchestration system that worked extremely well, before Kubernetes was cool.
As time passed by, that hot new stuff called Kubernetes catched fire and also Rancher Labs announced that the next version of their cluster management product (Rancher) would be based on Kubernetes.
Since “new is always better”, we decided that it’d be a good idea to let the newbie (me) learn about Kubernetes and come up with a good migration plan.
A few months (and a few hundred websites, docs, tutorials, etc.) later, we finally migrated our cluster from Rancher 1.6 to RKE (Rancher Kubernetes Engine) + Rancher 2.0 on top of it.
Now that production was running on Kubernetes, we thought about the next big move to pull off.
So far, the developers were in the great situation, that they could use docker-compose to deploy the whole application stack locally, with a pretty quick iteration cycle and then deploy that whole stack to the Rancher 1.6 cluster which used almost exactly the same technology (a bit adapted though, called rancher-compose).
With the migration of the production cluster to Kubernetes, we created quite a gap between the devs’ local development environment and the production environment, creating the need for duplicate configuration files and new bug vectors, e.g. by untested deployment manifests.
So finally, the problem ended up to be: “Let’s find a tool that is as simple and fast as docker-compose… but for Kubernetes”.
So we had quite a few requirements for such a tool. It had to…
- provide some of the convenience functionality that docker-compose brought, including
- volume mounts to support hot code reloading
- port forwarding to access webservices via
- have a functionality to use images built locally inside the cluster
- be FOSS
- be lightweight
- be easy to install and upgrade
- run on all common operating systems, but at least on Linux (mainly Debian/Ubuntu) and MacOS
- run fast
- be easy to create and decompose development environments
Back then, potential candidates included (I actually cannot remember all of the tools I checked out back then)
k3s(Rancher Labs, now part of CNCF)
Except for maybe
minikube, all of those tools were still pretty new and quickly evolving at that time.
Unfortunately, all of them had some drawbacks that made them unsuitable for us, e.g.
minikuberequired a virtual machine on all operating systems, which lead to slow startup times, making decompose-recreate cycles quite inconvenient
- nowadays, minikube is way faster and also provides a docker driver
kindwas still lacking some features (I guess that back then, the focus was mostly to run Kubernetes end-to-end tests in CI)
- the project now has a ton of amazing features, making it a great tool also for local development on Kubernetes
microk8swas only available on snap enabled systems
- later, the team behind the tool added support for running on other systems as well using a virtual machine
k3sonly runs on Linux and can only spin up a single cluster per node
- this is still valid today, as k3s is basically a Kubernetes distribution and not a tool to run Kubernetes (though it certainly makes it easier)
Wondering what a comparison would look like today?
I’m planning to create another blog post, comparing those options to k3d with their current feature state, so stay tuned!
Now while doing my research and giving all those tools a try, I stumbled upon a Twitter post by Rishabh Gupta, who wrote about his new tool called
k3s-in-docker that was a wrapper around the Docker CLI and the official K3s docker image to spin up single-node K3s clusters in Docker.
The combination of Docker, which was already available on all platforms and K3s as the lightweight and fast Kubernetes distribution, made
k3s-in-docker (written in Rust) another good candidate for the tool we were searching for.
While I gave it a try, I liked it a lot (might’ve been a bit biased though as I already liked K3s and many of the other projects coming from Rancher Labs), but it was still missing some essential features that we really wanted to have, e.g. volume mounts.
So I had a chat with Rishabh about my issue and also started to try and implement this feature myself.
But then it happened… impatience hit…
… I didn’t manage to wrap my head around Rust in around half a day and so I couldn’t implement that highly desirable feature.
I just learned some Go as part of my bachelor thesis and by working on some open-source tools. This and the learning, that almost everything in the Kubernetes ecosystem was (and is) written in Go, led me to thinking that it may be a good idea (and also a good training), to re-write
k3s-in-docker in Go.
So I created the new repository
k3d-go (iwilltry42/k3d-go now redirects to the new repo) and started coding away. Obviously, as I knew at least a little bit about Go and I already had some blueprint of the basic functionality (the commands to call to run k3s containers were the same after all), I pretty fast had an MVP ready that was able to create k3s clusters in docker with the volume mounts feature we wanted to have.
A few months earlier, I joined Twitter to stay informed about technology (as many brilliant minds are on there… amongst others 😁 ), so now I thought was my time to shine.. and brag about what I just accomplished, even though it was absolutely nothing special. I saw a good idea, saw a better fit for it in a different language (for learning and because of the surrounding ecosystem) and re-implemented it.
For refreshing my Go skills, I rewrote @zeerorg's (thanks for the original work!) k3s-in-docker project in Golang: https://t.co/mEIX5VwFkd— Thorsten Klein (@IWillTry42) April 3, 2019
It's a small helper CLI to make using @Rancher_Labs's k3s in docker a bit easier.
To my luck, Darren Shepherd, co-founder of Rancher Labs (now Chief Architect at SUSE) chose Twitter to be a good place to spend some time on that day.
When Darren saw Rishabh’s original blog post and my follow-up tweet about the rewrite in Go, he was super excited about the project building on top of his K3s and so he even created the very first PR.
Just shortly after my original Tweet, Darren, Rishabh and me got together in a group chat to talk about both versions of k3d and eventually decided that it would be best for the project to carry on the Go-based version and to move the repository to Rancher Lab’s space on GitHub, where more people would see (and expect) it: rancher/k3d.
We all loved the idea of pushing this project further, so we moved the repo and started to implement more features, add issues and fix bugs right away.
All of this came absolutely unexpected for me, but soon it showed that it was not only us three being excited about this new project as more and more people gave k3d a try, started creating issues and pull requests and were also just talking and chatting about it (e.g. at the KubeCon EU 2019).
I am happy to say that my repo iwilltry42/k3d got now transferred to @Rancher_Labs space on @github .— Thorsten Klein (@IWillTry42) April 12, 2019
You can now find it here: https://t.co/gYjwtLi0UY
Thanks to @zeerorg and @ibuildthecloud for collaborating on this despite 9h and 3h30m time difference :)
How It’s Going
Over the next few months (and even years by now), k3d was completely re-written from scratch, largely refactored more than once and gained lots of new features and improvements together with a growing user base and awesome community.
Still, k3d is a community project, that is really well supported by Rancher Labs (now known as SUSE Rancher), who e.g. provide the Drone CI infrastructure for testing and releasing k3d.
As of now (June 2021), k3d is approaching the major release v5.0.0 that will finally solve some issues that have already been around since the beginning in 2019 (like editing the cluster after it was created, e.g. to add new port mappings).
The main goal of k3d is to provide a fast and easy way of creating and managing decomposable Kubernetes clusters for local development.
As a side-goal, k3d is also a good candidate for throw-away test clusters in a CI system (and there are already real life examples for various CI systems out there).
k3d will continue to evolve and hopefully it further improves the life of developers, operators and everyone else interested in quickly spinning up a lightweight Kubernetes cluster.
Last but not least: as a community-driven project, we’re always happy about contributions, be it discussions, issues, pull-requests, blog posts, tutorials or videos, related projects or anything else that helps the community and the project!
Thanks everyone for contributing to improve k3d!
Some Stats on k3d-io/k3d
A Note About Me
To be able to spend more time on k3d, I got the amazing chance to work as a freelancer for Rancher Labs / SUSE Rancher as of March 2021, while still partly working with my original team at trivago. I am really grateful for getting the opportunity to work for those two amazing folks and being able to work on this great project, while still being able to make a living. Thank you!