.svg){kind=logo}
Orchestrating small boxes made easy
What if I told you that it was possible to deploy and orchestrate a fleet of physical servers as if they were a cloud provider, from the comfort of your own home (or office)?
What if you could do this even if those servers were basic Raspberry Pis, without any integrated management capability (like IPMI)?
It would be a cool thing to do in your homelab — but it would produce massive efficiencies in enterprise edge deployments, where small form-factor single board computers (SBCs) play such an important role in edge application deployments.
At Spectro Cloud we have a thriving homelab and OSS community chasing answers and building solutions to questions like the above every day. We love experimenting, particularly in the edge computing space.
And today I’m here to show you how you can remotely orchestrate SBC devices like RPi. If it has Power over Ethernet (PoE), we can make the magic happen.
Let’s get started.
Bare metal management: Canonical MAAS
There are a few puzzle pieces we need to assemble to pull this off.
The first is software that can act as a control plane for your fleet of servers, taking resource requests and determining what hardware can serve those requests. Canonical Metal as a Service (MAAS) perfectly fits this requirement.
MAAS supports server management tools such as IPMI, Redfish, OpenBMC, Proxmox, VMware, LXD and even AMT. If your physical hardware or virtual machine is supported, it can remotely control the server’s power state, the OS, and even configure its network.
Want to deploy Ubuntu to a single host, or a Kubernetes cluster to 100 Dell PowerEdge servers in 10 different server racks? MAAS can do it. Oh, and it’s free.
Power without the protocols
MAAS is fantastic, but it requires the machines it communicates with to support enterprise-grade remote management protocols such as IPMI.
And this is where we run into our first hurdle: MAAS needs to be able to power on or off a physical server to reboot it so that you can PXE boot and load an OS image.
If your hardware doesn’t have a baseboard management controller (BMC), how do you turn it on and off remotely?
- You can’t rely on tools installed after the OS is deployed, obviously!
- Wake on LAN can only “wake” a pc, not restart it or shut it off.
- You could manually push the power button every time MAAS requested. Indeed, MAAS does support this as the “Manual” Power Driver option, but it isn’t practical if you have to do this for hundreds of distributed computers.
What about Power over Ethernet though?
If we could remotely connect to a PoE switch and control the power state of each ethernet port individually, then not only could we control the power for each connected machine, we could also monitor if that machine is ready and able to communicate.
MAAS doesn’t have a Power Driver for PoE network gear though, does it?
Here’s the hook (the webhook, that is)
Webhooks are lightweight event-driven functions that you can use to communicate over http.
If you wanted to tell a server to run a task, you could send it a webhook command via URL such as https://myserver.local/api/run/job1. It can send you a response to this request with the status of job1.
MAAS has a Webhook Power Driver that allows you to talk to a device without a BMC and send it power on, power off, or status commands.
To use these commands, we would need an http server that could act as an intermediary between MAAS and the device we want to control. Lucky for us maaspower already exists and we don’t have to write this ourselves!
Maaspower supports a range of devices that we otherwise could not control from MAAS directly. With a bit of work, maaspower can be configured to support almost any kind of device, providing it has a way to manage it remotely via CLI, REST, or web interface. Maaspower supports command line tools, so if you can open a terminal and control your device from that terminal, there is a way to control it from MAAS automatically.
You could therefore connect a bunch of Raspberry Pis to a PoE switch, SSH to that switch and manually run commands to enable/disable PoE on each port… but then we’re just using the Manual Power Driver with extra steps.
If we can automate this connection then we have the last piece and we’ve solved the puzzle.
I have written a proof of concept command line tool for my Dell 5524p PoE switch, which demonstrates how you would do this. You can grab it here.
Time to plug in and turn on
So we know that to get true remote management for our SBCs like RPis, we can do it with MAAS, maaspower, and a PoE managed switch.
Let’s try it out.
We’ll assume you already have MAAS deployed on an Ubuntu host in your environment and you have synced an Arm64 Ubuntu 22.04 image to MAAS for use with RasPi.
We’re going to deploy maaspower to the same host for this example, following the instructions on the maaspower website.
Next, deploy maaspower-dellctl following the instructions available on github, configuring the values in poe.sh for your Dell PoE switch.
Next we need to set up a Raspberry Pi 4 to PoE boot off of our switch. You’ill need to burn the Raspberry Pi 4 UEFI Firmware to a microSD card, with instructions on how to do that available on github here.
We’re going to change a few things from the above guide for making the UEFI microSD card, as the script provided is a little out of date. The latest version of the UEFI Firmware is v1.37, so we should update the URL to point at that:
Further down is a command to format and partition the microSD card, and there’s an issue with it as it is now. The value on line 38 is too large and you may get an error message when you attempt to partition the disk:
Change the size from 2000000000:
To 20000000 or smaller:
Insert the microSD card in your Raspberry Pi, connect a keyboard and monitor and plug it into the PoE switch. Hit escape when prompted, because we need to perform a one-time configuration of the UEFI BIOS.
It’s a manual step, but it persists through reimaging so we should only have to do it once for as long as the box is in use.
First head to Device Manager > Raspberry Pi Configuration > Advanced Configuration and set Limit RAM to 3 GB to Disabled. Hit Y when prompted to return to the main menu.
Navigate to Boot Maintenance Manager > Boot Options > Change Boot Order and ensure that UEFI PXEv4 is the first item in the list. Hit Y to confirm and then Continue to boot to PXE.
Your Raspberry Pi should now boot and contact MAAS for PXE files. This will take a few minutes but you should see it display “Booting under MAAS direction”. Once it does you can monitor the status from the MAAS UI.
As soon as it shows up as a Machine we can configure it. First change the name to raspi + port number (raspi1 for PoE port 1) and hit Save:
Under Configuration the Power Type will default to Manual, change it to Webhook
We have some values to configure:
URI to power on the node: http://maas-controller.maas:5000/maaspower/raspi1/on
URI to power off the node: http://maas-controller.maas:5000/maaspower/raspi1/off
URI to query the nodes power status: http://maas-controller.maas:5000/maaspower/raspi1/query
Power User: test_user
Power Password: test_pass
These URIs are webhook commands! Each of them tells maaspower to perform a specific function on a specific PoE port and they’re how we’re able to automate control of our Raspberry PI from within MAAS.
Once you’ve finished configuring these values hit Save and then go back to Summary for this machine in MAAS. You should now be able to Check power for this machine.
From here you can Commission the machine and assign it to a Resource Pool. It is then ready for use. Repeat this process for every additional Pi.
The secret ingredient for edge orchestration
So, you created an automated fleet of Raspberry Pis that can have almost anything you want deployed to them, as if by magic. And you know how to control PoE devices from Canonical MAAS.
But how do you control MAAS?
Spectro Cloud Palette has a Cluster API provider for Canonical MAAS. With Palette driving MAAS, you can:
- Deploy a full Kubernetes-based software stack to your devices, supporting thousands of devices in parallel
- Cost-effectively automate operational tasks like patching/upgrades, configuration changes and observability, again in parallel across all your devices
- And do all this for your Kubernetes clusters in the cloud, data center and the edge, from a single consistent platform
You can use the existing Packs and resources it provides to deploy to amd64 machines in the cloud, at the edge or in your homelab, and you can even bring your own operating system for unique use cases!
