Why did I join Spectro Cloud? Because devex matters

New beginnings, same passion

This month I joined Spectro Cloud as a technology fellow. 

It's a bold new beginning for me. But it's also a natural continuation of a decade of work, all pursuing the same goal: helping software development teams build, deploy and manage applications more easily.

Let's jump back ten years and you'll see what I mean — and why I'm so pumped to join Spectro Cloud's mission.

In 2013 I began the most exciting part of my career (until now, that is!). That was when 1,200 of us from EMC and VMware came together to form Pivotal. VMware was incubating Cloud Foundry (CF) at that time and it was already a major part of my work in the EMC CTO office, so naturally I joined the Cloud Foundry team at Pivotal. 

I was blown away by the tech. Cloud Foundry:

• Was creating container images and then running them in Linux containers it created and managed. 
• Provided a declarative API and a container orchestration runtime similar to the Kubernetes replica set controller. 
• Was cloud agnostic and also ran on prem, effectively offering one of the first serverless (not FaaS) capabilities in the market. 

All of this before Kubernetes, and even Docker, existed. Oh yeah — the tech was cool! 

It’s all about the developer experience

But I've never been one to do tech just for tech’s sake — no, I need to see real user and business value, or at least have a gut feeling that it's there somewhere. 

And wow did Cloud Foundry ultimately bring value. Its developer platform provided self-service access to application teams so they could build software better… thereby building better software. 

This platform did such a good job of insulating users from infrastructure details that these application teams were able to break out of the old, slow software engineering practices, all while maintaining the security, compliance and SLAs that are non-negotiable in the enterprise. That’s the essence of good DX.

During this time I became really engaged with the DevOps community, earning my place on the DevOps 100 several years running, and serving on the programming committee for Gene Kim's DevOps Enterprise Summit. DevOps emerged as a formalism of the goal of doing software better, and Cloud Foundry did an excellent job of delivering on that goal.

And then, Kubernetes happened.

Kubernetes: the platform for platforms

DevOps teams were immediately interested in Kubernetes when it launched in 2015, but it wasn't until 2017 that Kubernetes became the de facto standard for container orchestration. 

I remember the first time I saw Kubernetes. I was at a conference and someone was doing a demo. I texted a colleague of mine who had already touched it and said "OMG!"; all he said back was "Yeah!". Kubernetes was simply revolutionary. 

I was lucky enough to start working with Kubernetes around that time, part of a joint Pivotal/VMware team that brought Pivotal Container Service (PKS — because at the time all the cool kids spelled "container" with a K 😉), a Kubernetes as a Service platform, to market.

“Container Service”. That name says a lot. At the time, Kubernetes was primarily regarded as a container orchestration tool. But the reason for my “OMG” was its revolutionary way of building systems through controllers.

You see, the architectural core of Kubernetes is that its functionality is implemented through a set of infinite loops that are responsible for bringing the state of the world in alignment with what ops team members want it to be. 

K8s shipped with one set of these infinite loops: those concerned with managing containers, which is why people see it as a container orchestration tool. 

That in itself is a huge achievement, but container orchestration was simply use case #1 for Kubernetes! Kubernetes can be extended with any number of these controllers or operators, and you can use them to manage just about anything. In the words of the Crossplane project, Kubernetes can be extended into a universal control plane.

At the end of 2019 I left Pivotal to join a company that was doing just that: extending Kubernetes. 

Weaveworks, which coined the term “GitOps”, had created the Flux operator, which allowed a Kubernetes cluster to constantly reconcile itself to a set of configurations stored in Git. This extends Kubernetes with continuous delivery functionality. 

And over the years as a member of the Technical Oversight Committee of the CNCF, I saw many more projects that adopted this tenet of continuous reconciliation (the infinite loop) to deliver a wide range of capabilities, such as gitops (ArgoCD, Flux), chaos engineering (Litmus), observability (Prometheus) and many more..

Platform engineering: the time is now for IDPs

In mid 2021 I stepped away from Kubernetes, or rather, developer platforms, to work in another tech space. When earlier this year I reengaged in the space I noticed two things had happened in that year and a half.

First, despite years of rumblings and even the existence of a smattering of projects (such as Backstage, Knative, Carvel, …), the Kubernetes community has — let’s be honest — failed to deliver a platform that abstracts away infrastructure details the way that CF did so effectively. 

Kubernetes — and the whole landscape of projects around it — remains fundamentally infrastructure technology, imposing a huge burden on the application teams who are trying to use it. 

‍

Second, I saw the emergence of an industry narrative around Platform Engineering and Internal Developer Platforms (IDPs). Eureka! 

Platform engineering is ultimately what made our customers successful with Pivotal Cloud Foundry (PCF) all those years ago.

We helped enterprises transform their "ops" teams into platform teams, who delivered PCF as a product to their internal customers, the developers/application teams. The fact that the whole industry is now talking about this tells me the time for developer platforms has truly come.

This is all goodness, but doesn't yet explain why I specifically came to Spectro Cloud. 

Why Spectro Cloud? Because in IDPs, one size does not fit all

Kubernetes is a platform. But it's not a developer/app platform, rather it is a platform for building platforms.  

As described in the Kubernetes docs, "Kubernetes provides the building blocks for building developer platforms, but preserves user choice and flexibility where it is important." 

The latter half of this sentence is really important. 

You see, Cloud Foundry was a fantastic developer platform — it abstracted away details that weren't germane to the app teams, and implemented strict guardrails to ensure security, compliance and all of those other enterprise concerns along the golden paths to software delivery.

It was unapologetically opinionated. If an organization was good with those opinions (and many were), they were very successful with their Cloud Foundry use. 

But if they came with differing opinions (and there were many of those, too), well, they were mostly out of luck.

Here's the punch line. There can be no single, one size fits all developer platform. 

Every enterprise engineering team has its unique business needs, its own customer base, its own development needs, its own controls. Each organization needs THEIR platform to enable developers. 

Years ago we thought teams who started building internal developer platforms were crazy — IDPs are built by the FAANG and that’s it. We saw many others try, and virtually all failed. 

But I now believe that if we give organizations the right tools, and great foundations like Kubernetes, their platform engineers will be able to build and maintain the developer platforms they need.

The other day I was chatting with Tenry and Saad and I asked them, "this concept of a Cluster Profile, has that been a central part of Spectro Cloud from the beginning?" and the answer was a resounding, and emphatic, "yes". Kubernetes and the broader ecosystem provide a lot of building blocks, but we need tools to assemble them, and maintain those assemblies.

THAT is why I have joined Spectro Cloud!

Composability is the foundation of choice

Spectro Cloud started with this as a core tenet: That organizations need help delivering internal developer platforms uniquely suited to their business. 

As a starting point the team has built out two things:

1. A set of building blocks that can be used to construct what you need in your platform. We call these packs (the Lego bricks).
2. A way of creating and managing compositions. These are the aforementioned Cluster Profiles (the Lego building instructions)

‍

‍

Organizations use these things to extend Kubernetes, to construct exactly the platforms they need. And through them, Palette aims to give you the best of both worlds: true simplicity, automation and abstraction, while still giving you flexibility and freedom of choice.

So far, Spectro Cloud’s engineers have been laying foundations in a number of areas, starting with Kubernetes cluster management and expanding out to the edge. Now we are ready to go all the way to the developer — to the IDP.

So here I am: here to help Spectro Cloud build on the next generation of developer platforms, nurturing the green shoots of Palette Dev Engine to optimize user experience for developer productivity. I have seen first hand what a good developer experience can result in, and now I want to take that to the next level. And as I do so, I look forward to working closely with you.  

In my time at Pivotal and Weaveworks I had the opportunity to work directly with dozens of large organizations, helping them to revamp the way they built and managed their digital offerings. I knew I wanted to be back out there co-inventing with these driven, creative and gutsy folks. 

So all that’s left to say is this: Come say hello. Join me in the Spectro Cloud community Slack, or throw me a curveball question on my forthcoming CNCF livestream (details coming soon!). Let’s build the future of devex together.