A developer’s guide to deploying applications in air-gapped Kubernetes

What do you do when you can’t grab images?

Airgapped environments are much more common than you might think. They’re commonly used in military and defense systems, financial services, medical facilities, government agencies, and high-security corporate settings.

The air gap is — or should be — total. The environment is cut off from the public internet and other external networks. Any access to the outside world is a risk to security and privacy and a breach of policy and potentially regulation too.

Other sites are effectively airgapped by default, and have little or no access to the outside world whether due to flaky rural broadband, interference or remote locations.

Let’s take a field engineer in one such air-gapped location: a government R&D laboratory. He wants to get a simple app working in the lab. On his internet-connected machines or a conventional connected cloud or data center cluster, installing an application would be as easy as pulling and running a container image from DockerHub.

Since he works in an air-gapped lab, he knows he cannot connect to registries or repositories to pull down all the images, dependencies and configuration files needed. He can't use a CI/CD pipeline to keep his app up to date and all his software has to be signed and scanned. He needs to package up and manually carry over everything his application needs to work on the local environment. Let’s explore how we can help him achieve this.

The on-site setup: don’t forget the cluster

In the lab, our engineer has the following setup:

A Kubernetes cluster to run applications
A Harbor repository for storing the necessary secure images locally
A USB or hard drive to bring in zip files
A virtual machine (jumphost) to connect to the Kubernetes cluster

Note that bootstrapping a Kubernetes cluster in an airgap environment is itself no small feat. Tools like clusterctl can help set up DIY clusters in data centers managed by VMware or MAAS. For more complex or production-grade clusters, solutions such as those offered by Spectro Cloud offer options for air-gapped deployments across various cloud environments. But for this tutorial, we're just going to focus on bringing up the application.

Our task is to move a simple application into this secure environment, ensuring it’s set up correctly and ready for use. This process will help us understand how applications are created, updated, and deployed in an air-gapped setting.

To deploy an application into an air-gapped environment, we first need to gather all the necessary artifacts from the internet-connected side.

Collect everything you need on the internet-connected side

To get the application running, you’ll need the Kubernetes manifest file associated with it. In the real world, we could use tools like Zarf to get our application packaged. Technologies like Zarf allow declarative creation and distribution of software into airgapped environments by providing a way to package software that is repeatable, secure, and reliable.

For the sake of simplicity, you can find the manifest file for this tutorial below. This file includes all the configuration details required to deploy a simple gamebox app on any Kubernetes cluster. Of course, in the real world our engineer is unlikely to be playing video games on the job, but the process is the same!

manifest.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: game
spec:
  selector:
    matchLabels:
      app: game
  template:
    metadata:
      labels:
        app: game
    spec:
      containers:
        - name: multi-game
          image: "defenseunicorns/zarf-game:multi-tile-dark"
          ports:
            - name: http
              containerPort: 8000
              protocol: TCP
          resources:
            requests:
              memory: "32Mi"
              cpu: "50m"
            limits:
              memory: "128Mi"
              cpu: "250m"
---
apiVersion: v1
kind: Service
metadata:
  name: doom
  annotations:
    zarf.dev/connect-description: "Play doom!!!"
    zarf.dev/connect-url: "?doom"
  labels:
    # Enables "zarf connect doom"
    zarf.dev/connect-name: doom
spec:
  selector:
    app: game
  ports:
    - name: http
      port: 8000
      protocol: TCP
      targetPort: 8000
---
apiVersion: v1
kind: Service
metadata:
  name: game
  annotations:
    zarf.dev/connect-description: "Play some old dos games 🦄"
  labels:
    # Enables "zarf connect games"
    zarf.dev/connect-name: games
spec:
  selector:
    app: game
  ports:
    - name: http
      port: 8000
      protocol: TCP
      targetPort: 8000

Next, we’ll go through the steps to apply this manifest and get the gamebox app up and running in the air-gapped environment.

Let’s inspect the manifest to see which docker image it points to:

As a first step let’s download this image into a tar file. I did using a docker save on my local machine.

docker pull gcr.io/spectro-dev-public/devx/spectro-gamebox
docker save -o gamebox.tar gcr.io/spectro-dev-public/devx/spectro-gamebox

Now I have two files in the folder:

ls
gamebox.tar  gamebox.yaml

Bundling the resources into a single zip file:

zip -r  gamebox.zip gamebox/

Move the artifacts into the secure environment for deployment

air-gap environments illustration with a developer on site

Now let’s move the zip into the airgapped environment the old fashioned way: by putting it on a USB drive and using the sneakernet. Connect it to the local machine and unzip it on the local machine OS.

unzip gamebox.zip

First load the tar file into the local docker using the docker load command

docker load < gamebox.tar

Tag the image into the local registry location

docker tag gcr.io/spectro-dev-public/devx/spectro-gamebox:latest demo.spectrocloud.dev/games/spectro-gamebox:latest

Push the image to the local repo

docker push demo.spectrocloud.dev/games/spectro-gamebox:latest

You can always use tools like image-swap to swap images automatically, or you need to edit the manifest file to change the image URL. For simplicity we’ll just change the URL in the gamebox.yaml manifest file.

Deploy the app in the airgapped environment

And finally apply the manifest file using the below command:

kubectl apply -f gamebox.yaml

You can see the deployment, pod and service up and running as shown below

code for donwloading gamebox and deploying in an air-gapped environment

And there you have it: the gamebox app is up and running at http://node_ip:31000 port

showing how the app starts running in airgapped

What about updates and airgap at scale?

This approach ensures that the gamebox application, or any other software, can be efficiently deployed within a secure, air-gapped environment. For future updates to the application, the process remains the same: package the updated artifacts, upload them to the registry, and apply the updated manifest.

This process works well for a simple app on a single cluster that already exists. But how often is that really the case in the world of enterprise or public sector?

Many of our air gap customers have multiple applications that need to run on many clusters, both within the same air gap site and in multiple sites. In that scenario, you’re looking at multiple repetitions of these manual steps. CI/CD pipeline tools like Argo CD and Flux CD would really help in automating the process and reducing redundancy. For more complex needs, a platform like Palette can help.

For further reading and experimenting, here are other airgap resources:

Tags:

How to

Developers

Concepts

Networking

Operations