.svg){kind=logo}
Now we’re all using microservices on Kubernetes, there are no software development problems left to solve. We’ve achieved peak efficiency. Our developers are highly productive and not only deliver ‘just in time’, but with amazing quality.
Or maybe not. Back in the real world, there’s plenty of room for improvement, with new tools and best practices that we can adopt.
Let me share one with you now: trunk-based development.
What is trunk-based development?
In short, and with injustice to the practice, trunk-based development is an approach to version control systems where you leave work every day having committed all your changes to the trunk. The next day, you start with a clean, fresh environment and do it all over again.
To quote from the official site:
“Trunk-Based Development is a key enabler of Continuous Integration (CI) and by extension Continuous Delivery (CD). When individuals on a team are committing their changes to the trunk multiple times a day it becomes easy to satisfy the core requirement of Continuous Integration that all team members commit to trunk at least once every 24 hours. This ensures the codebase is always releasable on demand and helps to make Continuous Delivery a reality.”
No branches, no merge hell
Trunk-based development limits drift in code changes by avoiding the use of branches. This matters because you can burn weeks merging all the other changes that have happened while that feature branch remained separated from trunk (aka Merge Hell)
Protecting production with feature flags
Now I know what you’re saying: it can take weeks to get a new feature created. In the meantime, who in their right mind would allow unfinished code to find its way to a production environment via a unified codebase?
There is an answer: feature flags. When you start a new feature, that feature should not only be tracked in a story or task, but also with the use of a feature flag (or several).
The net effect is that your code changes should be hidden/disabled until they are ready to be exposed to your end users, reducing the risk of any bugs.
A clean build environment
A key concept in trunk-based development comes from best practices in the CI space: the idea that you should have repeatable builds.
Every build that the CI system does should be done with a brand new, clean build agent and environment, effectively ensuring that the only thing that is different from one build to the next are the actual code changes.
Trunk-based development shares the same premise: you should start your work from a known functional state. This is invaluable to development because it ensures that pre-commit code changes will actually survive the CI pipeline.
So a fresh checkout from trunk is the first place to start. But now you start writing your code. Like any good developer, you want to see and test your code work in real time, without waiting for a ticket. You need an environment to deploy to!
Testing in the world of microservices: it’s messy
Remember, you’re writing a microservice, so that implies you need some other services running somewhere for yours to interact with.
Many shops will simply set up a shared development Kubernetes cluster where all the services are running, so each developer can test their changes as they make them.
But there’s a problem: other developers are continually changing their services too, meaning you’re testing in an unpredictable setup. You’ve likely spent the past four days troubleshooting a problem that a teammate caused by tinkering with an API call signature.
Namespaces aren’t the solution
Another solution is to give each developer their own namespace in the development cluster so that they can stand up the latest version of everything their service needs to interact with.
This is actually a great place for many people to be. However, you’ve still got to deal with configuration drift from leaving a cluster running long-term (it’s no longer pristine). And you’ve still got to keep up with all the other things that are going to production which are in sync with trunk.
There’s also the fact that our services are often designed to run in separate namespaces. So we have to test other cluster changes such as role bindings, network policies, and the myriad things that can result from needing higher level access to a cluster than what operations say I’m allowed to do.
Virtual clusters are a piece of the puzzle
The real solution? Virtual Kubernetes clusters. If you haven’t already heard about virtual clusters and the vCluster project, let me enlighten you. This is an open source project that allows you to create virtual clusters on a host Kubernetes cluster, much like you would run a VM on a real machine.
vCluster enables you to easily spin up a short-lived Kubernetes cluster and get full admin control over that cluster so that you can test out all the services, settings, role bindings, etc that you need to build your application (or your specific part of it anyway). And just as with your fresh code checkout, you can create a fresh cluster every day, and tear it down every day.
But when you fire up that cluster each morning, you still need to get it into a ‘ready to use’ state, so you can focus on your code.
How great would it be if you could make a single CLI call to spin up a virtual cluster that has all the things you need to be productive, including a fresh git clone?
Sure, you can use Terraform for that, but there is a faster and easier way.
Spectro Cloud Palette has this thing called a Cluster Profile. It’s basically a blueprint for what a cluster should have and how it should be configured. Palette uses the Cluster Profile to declaratively build out that configuration, then actively prevents configuration drift. You can spin up a virtual cluster and Palette will apply the Cluster Profile to it, resulting in a fully baked cluster that’s ready to use.
Speed, safety and isolation
Virtual clusters come up fast. Fast enough that you can refresh the cluster several times over the day without it slowing you down. It’s equally easy for you to tear down your development cluster when you’re done with it, all in a self-service manner.
It is absolutely amazing to be able to have a clean, predictable and working environment every time you sit down to work on an application. But more than that: to do so with the knowledge that the cluster you are using was set up using company standards. You don’t have to worry about discovering, after your tenth CI build, that you broke a policy.
You can take the use of virtual clusters even further. You can set up a development pod that has a container for your IDE and a build container that runs your app as you make changes. Talk about instant gratification. Now you know that your work will survive the CI pipeline and you can finally go home on time.
You can also leverage this setup to help with peer review. The CI pipeline can create clusters on the fly to ensure unit testing is done in isolation (a key requirement if you want to effectively troubleshoot problems) before doing full integration testing and bug fixes.
Let’s see it in action — and you can follow along
Time to try the concepts out for real. If you don’t have a Palette account, you can still follow along, and all the examples below are available in this GitHub organization.
Meet the application components
This example will use a generic API server, with a Node application front end, along with an IDE, all running in a virtual cluster.
In our example the API service was created by someone else and should be accessed in the same way as if it were running in production. That is to say, our ingress will route /api traffic to this service while the front end loads from the base URL.
Keep in mind that this model could work equally well for a service that needs to interact with another service in the same cluster. The goal here is to ensure that the API server is both not the one running in production, and not something that someone else might change while working on our own code changes.
containers:
- image: vad1mo/hello-world-rest
name: core-rest-server
vad1mo/hello-world-rest - A simple REST server that reflects wherever you send it.
You will also need to create a service and optionally an ingress. My full example is here. Once you have this up and running a few simple curl commands can be used to test it.
Choosing the IDE
Most developers would use a local IDE and run their apps locally to test them. The downside is that personal desktops tend to become cluttered. It can be hard to know for sure if the app would work properly in prod, or just on your desktop.
Remember, part of the idea behind trunk-based development is that every time you sit down you are working from a known good state. Personal workstations are notoriously unhelpful in this regard!
So here we’re going to use an in-cluster IDE with an in-cluster/live instance of our app. For this I’m using Visual Studio Code Server and its browser-based IDE. I’m also going to run a node container that shares the workspace with the code server. This node container will install any dependencies at startup and immediately go into dev server mode, serving up the latest state of my application.
containers:
- image: node:lts
name: node
- image: lscr.io/linuxserver/code-server:latest
name: code-server
Development Pod: https://github.com/tbd-with-palette/development-pod/tree/main
Triggering the virtual cluster with Palette’s CLI
Here is where the magic starts to happen. I have all this defined in a Cluster Profile in Palette. When I use the Palette CLI to spin up my development virtual cluster, it will use my Cluster Profile to create the API server, code server and node pod. This will also automatically invoke a fresh git clone from trunk and start my development node server.
$ bin/palette pde virtual-cluster create --name dan-dev-cluster
After a short wait I’ll be able to dive right into working on my next feature, and deal with any small merge issues before going home for the day.
Experience a moment of Zen
Adopting trunk-based development with virtual Kubernetes clusters can really contribute to the well-being and job satisfaction of your developers, giving you a fast and predictable way to code, test and deploy your microservice applications without mess or merge hells.
You can get there the OSS way, rolling your own with vCluster, Terraform and manual processes. But a platform like Palette from Spectro Cloud makes it easier, faster and safer. If you’d like to try it for yourself, book a demo here.
