There’s no platform team. No infrastructure review board. No second pair of eyes on a Terraform apply. When I push a change to the runner infrastructure, I’m the one who wrote it, reviewed it, approved it, and deployed it. If something goes wrong in production, I’m the one who gets the Slack notification and I’m the one who fixes it.
This post is about the practices and guardrails that make that work without constant anxiety. None of this is groundbreaking — it’s mostly just discipline and good defaults. But the specific combination matters, and I think it’s useful for anyone else running infrastructure as a solo operator or a very small team.
The most important decision was treating dev and prod as completely separate environments from the start, not just logically but in every layer of the stack.
Terraform workspaces keep the state files isolated. The dev workspace and the prod workspace each have their own state, their own resource naming, and their own configuration. A terraform destroy in dev can’t touch prod resources because they don’t exist in the same state file.
Each environment has its own tracked tfvars file with explicit configuration. Dev uses a t3.xlarge with a 100GB root volume. Prod uses an m7i.xlarge with the default 300GB. Dev runners have the label sunrise-dev. Prod runners have the label sunrise. This means a dev job physically cannot be picked up by a prod runner, and vice versa. The labels are the isolation boundary at the GitHub level, and the Terraform workspaces are the isolation boundary at the AWS level.
Even the AMIs are separated. Dev bakes from the dev instance and writes to a dev SSM parameter. Prod reads from a different SSM parameter that only gets updated through an explicit promotion step. There is no path where a dev AMI accidentally ends up in production without someone intentionally promoting it.
I also have a third Terraform workspace — ci-iam — that owns the account-global resources like the GitHub OIDC provider and the CI IAM role. These are resources that both dev and prod depend on, but that shouldn’t live in either workspace. Having them in their own workspace means I can update IAM policies without touching runner infrastructure, and I never risk accidentally duplicating the OIDC provider across environments.
The CI pipeline authenticates to AWS using GitHub’s OIDC integration. There are no AWS access keys stored in GitHub secrets. The GitHub Actions runner assumes an IAM role via OIDC, and the trust policy is scoped to this specific repository.
This matters for a solo operator because credential rotation is one of those tasks that’s easy to forget about and catastrophic to get wrong. With OIDC, there’s nothing to rotate. The trust relationship is between GitHub’s identity provider and the AWS IAM role, and it’s verified on every run.
The GitHub PAT that the runners use to register with GitHub is stored in SSM Parameter Store as a SecureString. It’s the one long-lived credential in the system, and it’s the only thing I’d need to rotate manually. Everything else is ephemeral.
The weekly pipeline runs end-to-end on Sunday: bake the dev AMI, promote it to prod, apply Terraform against prod, roll the runner instances, and notify Slack. But the key design choice is that each stage is a separate job with an explicit dependency on the previous one succeeding.
If the dev bake fails, promotion doesn’t happen. If promotion fails, prod apply doesn’t happen. If the apply fails, the runners don’t get rolled. At no point can a downstream stage execute if an upstream stage failed. This sounds obvious, but it’s easy to get wrong with GitHub Actions — the default behaviour for dependent jobs is to run unless you explicitly check the outcome of the dependency.
The manual fallback is there too. If something needs to go to prod outside the Sunday cycle, I can trigger the promote_ami action from the prod workflow UI, then run an apply. But the default path is the automated pipeline, which means prod changes are consistent, predictable, and always preceded by a week of dev validation.
This has caught real issues. There was a week where the dev bake succeeded but the AMI had a subtle problem with the LFS cleanup that only showed up under specific repository configurations. Because dev was running that AMI for a week before promotion, we caught it during normal dev CI usage. If I’d been baking directly into prod, that would have been a production incident.
Not every change can be deployed through AMI promotion. When the runner bootstrap code changes — the Python runtime that gets embedded in the EC2 user-data — the fix lives in the launch template. An AMI promotion won’t pick it up because the existing instances are already running. You need to replace the instances.
The rollout module handles this automatically after a prod apply. It surges the ASG by one instance, waits for the new instance to be fully ready — InService in the ASG, SSM agent online, runner services active in systemd, and registered with GitHub — and only then drains and terminates the old instance. If the drain fails for any reason, it leaves the extra instance running. Availability over perfect cleanup.
This was one of the later additions to the system. Before this, a bootstrap code change in prod meant manually terminating the instance and waiting for the ASG to replace it. That’s a few minutes of downtime where no runner is available, which is fine at 2am but not great if someone triggers it during working hours. The automated rollout means I can merge to main, trigger a prod apply, and the instance replacement happens safely without me babysitting it.
The CI IAM role has three managed policies, each scoped to exactly what it needs. The S3 backend policy can read and write the Terraform state bucket. The AMI cleanup policy can deregister and delete snapshots, but only for resources tagged with the dev environment — it physically cannot delete a prod AMI. The workload policy covers EC2, ASG, Lambda, and the other resources the runner infrastructure needs, with ARN patterns scoped to the project’s naming convention.
I won’t pretend this was easy to set up. The initial deployment involved a lot of “apply, see what fails, add the missing permission, apply again.” And the policies had to be refactored when they started hitting the AWS managed policy size limit. But the result is a CI role that can only affect the resources it’s supposed to affect, which is important when that CI role is the primary deployment mechanism and there’s nobody reviewing the applies in real time.
One specific decision that paid off: the AMI cleanup policy is explicitly restricted to dev-tagged resources. The safety cap in the cleanup code (from Part 2) is the application-level guard against deleting too many AMIs. The IAM policy is the infrastructure-level guard against deleting the wrong ones. Two layers of protection for the same destructive operation.
The CI logic is structured as a shared workflow that both dev and prod call with different inputs. The shared workflow handles init, format checking, validation, planning, and apply. The environment-specific workflows handle the inputs — which workspace, which var file, which actions are available.
This means there’s exactly one place where the Terraform plan/apply logic lives. If I fix a CI bug, both environments get the fix. If I add a new validation step, both environments get the step. The alternative — copying workflow YAML between dev and prod files — is how you end up with environments that slowly drift apart until something works in dev and breaks in prod for reasons that take an hour to debug.
The Sunday pipeline sends a Slack notification when it completes, regardless of whether it succeeded or failed. The message includes the workflow name, the repository, the workspace, and a link to the run.
This is the simplest form of observability, and for a solo operator it’s one of the most important. If the Sunday bake fails, I find out in Slack on Sunday evening, not on Monday morning when someone complains that their build is slow because the AMI is a week out of date. If it succeeds, I get a quiet confirmation that the system is healthy and I don’t need to think about it.
The notifications only fire on scheduled runs. Manual runs don’t notify, because if I’m running something manually I’m already watching it.
If I had to summarize what makes solo infrastructure management work, it would be this: make the safe path the easy path.
Promoting to prod should be easier through the pipeline than around it. Rolling instances should happen automatically, not manually. Destructive operations should have safety caps. Credentials should be ephemeral. Environments should be isolated by default, not by convention.
None of these are novel ideas. But the difference between knowing them and implementing them is the difference between a system you’re confident in and a system you’re afraid to touch. Building this infrastructure over the past 7 months — from a single bash script to the system described across these 5 posts — the thing I’m most satisfied with isn’t any individual feature. It’s that I can push a change on a Wednesday afternoon and not spend the rest of the day worrying about whether I just broke CI for the whole team.
This is Part 5 of a series on building production-grade self-hosted GitHub Actions runners at a robotics startup.
In Part 2 I mentioned the November 6th marathon — 22 commits in a day fixing AMI cleanup edge cases, all written in inline bash inside GitHub Actions workflows. I said I knew I’d need to rewrite it, but I had other priorities. That was true. It’s also true that by the time I finally did the rewrite, the bash had grown into something genuinely hard to maintain.
This post is about what that migration looked like, why it mattered more than I expected, and what the Python runtime actually does.
The first version of the runner infrastructure was mostly bash. The EC2 user-data script that bootstraps a new instance was a shell script. The AMI baking logic was inline bash in the GitHub Actions workflow. The pre-job cleanup hooks were bash. Even the runner registration and deregistration was handled by shell commands piped together with && and ||.
For a while this worked fine. Shell scripts are great for stringing together AWS CLI calls and systemd commands, and when the system was simple — one instance, one runner process, no autoscaling — there wasn’t enough complexity to justify anything else.
The problems crept in gradually. The first sign was the pre-job hook. We run persistent runners, which means the same runner picks up job after job without being destroyed between runs. That’s great for performance, but it means the workspace directories accumulate state between jobs. Git lock files from crashed builds, stale LFS hooks from repos that used to use LFS but don’t anymore, repo-local git config that bleeds across unrelated jobs.
The hook started as a two-line script that deleted .git/index.lock before each job. Then it needed to handle nested lock files. Then it needed to clean up LFS hooks that were causing checkout failures on runners that didn’t have git-lfs installed. Then it needed to scrub repo-local git config. Each addition was a few more lines of bash, and each one had to be careful about paths, error handling, and not breaking the cases it wasn’t trying to fix.
By the time the hook was “done” it had been through 5 iterations across multiple PRs, and it was still fragile. A missing quote here, a glob that expanded wrong there. Every fix felt like it might introduce a new edge case.
The real breaking point wasn’t a single incident. It was the accumulation of several things happening around the same time.
The AMI baking pipeline had grown from a simple “stop instance, create image” workflow to a multi-step orchestration with detach, patch, pre-pull, cleanup, snapshot, roll forward, and smoke test. All in bash. With error handling that was either missing or wrong.
The user-data bootstrap script had evolved to support multiple concurrent runner processes per instance, with systemd service templating, signal handling for graceful shutdown, and runner registration token lifecycle management. In bash.
And I was hitting POSIX compatibility issues. The EC2 instances run commands via SSM, and SSM’s AWS-RunShellScript document uses /bin/sh, not /bin/bash. Features I was relying on — pipefail, arrays, certain parameter expansion syntax — either didn’t work or behaved differently. I’d fix something locally, push it, and watch it fail in a completely different way on the instance.
The tipping point was when I needed to add autoscaling support. The Lambda metric publisher needed to be Python anyway (it’s a Lambda function), and looking at the codebase, I realized I was about to have the autoscaling logic in Python, the baking logic in bash, the bootstrap in bash, and the cleanup in a mix of both. Three different languages and paradigms for what was essentially one system.
So I carved out a week and rewrote everything.
The core of the system is now a Python package called runner_ops. It has a CLI entrypoint with subcommands for each operational task, and a set of modules that handle specific domains.
The bootstrap runtime is the big one — about 850 lines that get base64-encoded and embedded directly into the EC2 user-data via cloud-init. When an instance boots, the cloud-init script decodes the Python runtime, writes it to disk, and runs it. The runtime handles everything the bash used to do and more:
It queries the EC2 instance metadata service to figure out its own instance ID and region. It downloads and extracts the GitHub Actions runner archive. It creates systemd service units for however many concurrent runner processes the instance is configured to run — if the config says 4 concurrent runners, it generates 4 separate services, each with its own working directory and registration. It handles runner token registration with GitHub, including generating unique runner names based on the instance ID and slot number so there are no naming conflicts across the fleet.
The persistent runner loop is where the bash-to-Python difference matters most. Each runner process runs in a loop — register, start the listener, wait for it to exit, clean up, re-register. In bash, the signal handling for graceful shutdown was a mess of trap statements that sometimes worked. In Python, it’s a signal handler that sets a flag, and the loop checks the flag between iterations. If a SIGTERM arrives while a job is running, the runner finishes the current job and then exits cleanly instead of being killed mid-build.
The pre-job hook went from a brittle bash script to a Python function that systematically walks the workspace directories and cleans up git lock files, LFS hooks, LFS config, and repo-local git configuration. It logs what it finds and what it removes, which made debugging checkout failures significantly easier. The bash version would silently succeed or silently fail. The Python version tells you exactly what state it found.
The AMI baking module is the 900-line orchestrator from Part 2. The finally block alone justified the rewrite — it ensures the ASG min_size gets restored and detached instances get terminated even if the bake fails halfway through. In bash, there was no equivalent. If the script failed after detaching the instance but before the cleanup, I’d have a dangling EC2 instance accumulating charges until I noticed.
The AMI cleanup module is the safety-capped deletion logic. The autoscaling metric publisher is the Lambda from Part 3. The notification module handles Slack webhook calls for pipeline status. The rollout module handles zero-downtime instance replacement after a prod apply.
There was one genuinely tricky part of the migration: EC2 user-data has a 16KB size limit. The bash bootstrap was already pushing against it, and 850 lines of Python is significantly larger than the shell version it replaced.
The solution was compression. The Terraform template base64-encodes the Python runtime and a JSON config blob. The cloud-init script — which is still a thin shell wrapper, because it has to be — decodes the base64, decompresses it, writes the Python files to disk, and starts a systemd service that calls the Python entrypoint.
The shell wrapper is about 60 lines now. Its only job is to unpack the Python runtime and hand off control. All the actual logic lives in the Python, which means all the actual logic is testable.
The testability is what made the biggest difference in practice. Every function in runner_ops that computes something has unit tests. The cleanup plan logic, the shell command builders, the tag verification, the polling utilities, the metric publisher. The AMI baking module has tests for the error handling paths that were completely untested when they were bash.
When I added the LFS cleanup in March — the fix for checkout failures on persistent runners — I wrote the test first, implemented the cleanup function, and verified it worked before deploying. In the bash era, I would have written the fix, deployed it to dev, triggered a bunch of jobs, and hoped for the best. Then probably spent another 2-3 iterations fixing edge cases I didn’t think of.
The other thing that mattered was the CLI. Having runner_ops as a proper CLI tool with subcommands meant the GitHub Actions workflows became thin wrappers:
- name: Bake and roll dev AMI
run: python -m runner_ops.cli bake-dev-ami --workspace dev ...
The workflow doesn’t contain logic anymore. It passes configuration to the CLI, and the CLI does the work. If something goes wrong, I can reproduce it locally by running the same command with the same arguments. That was never possible with inline bash workflows where the logic was split across workflow YAML, shell scripts, and SSM commands.
I should have started with Python. Not because bash is bad — it’s great for simple automation. But the moment I had multi-step orchestration with error handling and cleanup requirements, bash was the wrong tool. I knew this intellectually. I just kept thinking “I’ll rewrite it after this next feature” and then the next feature added more bash.
The migration itself took about a week of focused work. That’s not nothing, but it was less painful than I expected because the system’s behaviour was well-understood by that point. I wasn’t designing something new — I was translating something that already worked (mostly) into a language that could handle the edge cases properly.
If I were starting this project over, I’d write the cloud-init wrapper in shell — because you have to — and everything else in Python from day one. The initial velocity of bash isn’t worth the maintenance cost once the system gets beyond a few hundred lines.
The tooling is in good shape now, but there’s one more piece of the story: how do you safely manage all of this as a single engineer? Dev/prod isolation, deployment gates, automated rollouts, and the practices that prevent a solo operator from accidentally breaking production. That’s Part 5.
This is Part 4 of a series on building production-grade self-hosted GitHub Actions runners at a robotics startup.
The problem was burst load. When multiple developers push at the same time, or someone triggers a batch of integration runs, all 4 runner slots fill up and everything else queues. During one busy week we saw jobs waiting over an hour and a half for a free slot. CPU was hitting 100% on the instance and the queue just kept growing.
We needed autoscaling. And the standard advice for self-hosted GitHub runners is to use webhooks — listen for workflow_job events, spin up instances when jobs are queued, tear them down when they finish. GitHub’s own docs recommend this pattern, and tools like actions-runner-controller are built around it.
We went a different direction.
Webhooks are the obvious answer, and for larger teams with existing infrastructure they’re probably the right one. But for us, a small team with no public-facing endpoints and one DevOps engineer, the webhook approach had some problems.
First, you need something to receive the webhooks. That means either a public endpoint with authentication and TLS, or a more complex setup with API Gateway or a queue in front of it. Either way you’re maintaining infrastructure whose sole job is to receive HTTP callbacks from GitHub.
Second, webhook delivery isn’t guaranteed. GitHub has retry logic, but if your receiver is down during a burst, you miss the scaling signal entirely. Now you need to handle missed webhooks, which means you need a fallback mechanism anyway — at which point you’re maintaining two scaling signals instead of one.
Third, the coupling is tight. Your scaling logic is now driven by GitHub’s event schema. If GitHub changes the webhook payload, or if delivery latency spikes, your scaling reacts differently. You’re building reliability on top of someone else’s delivery pipeline.
What if we just asked GitHub “how busy are you right now?” on a schedule and scaled from there?
The idea is simple. A Lambda function runs every minute, calls the GitHub API to check how many runners are currently busy, and publishes that number as a CloudWatch metric. Standard CloudWatch alarms watch the metric and trigger ASG scaling policies when the count crosses a threshold.
That’s the whole architecture. No public endpoints, no webhook receivers, no event-driven complexity. Just a scheduled function that publishes a number, and AWS does the rest.
The Lambda itself is about 170 lines of Python. It fetches a GitHub PAT from SSM Parameter Store, paginates through the org runners API filtering by label, counts how many are busy, and publishes the count to CloudWatch with dimensions for the ASG name and runner label. EventBridge triggers it every minute.
The scaling logic is two CloudWatch alarms and two ASG policies. When the busy runner count hits 4 — meaning all slots on our instance are full — the scale-out alarm fires and adds one instance. When the count drops to 0 and stays there for 5 consecutive minutes, the scale-in alarm fires and removes one. SimpleScaling policies with a 60 second cooldown on scale-out and 300 seconds on scale-in.
Before writing any code I wanted to know whether this was actually worth building. I pulled 14 days of GitHub Actions history from the API and built a queue-wait report — looking at how often all 4 runner slots were occupied, how long jobs were waiting, and what the actual queuing pattern looked like.
The data made it pretty clear. There were regular bursts where all 4 slots were full and jobs were backing up. Not constantly, but often enough that developers were noticing and occasionally scheduling their pushes around each other to avoid the queue.
I also ran cost projections for the additional instance. With a scale-out threshold of 4 and a 5 minute scale-in window, the extra instance would run for maybe a few hours per month during peak periods. The projected cost was under $3/month. For context, the time developers were losing to queue waits was worth significantly more than that.
The report made the decision easy. This wasn’t speculative infrastructure work — there was a concrete problem with measurable impact, and the fix was cheap.
The Lambda handler is straightforward. The interesting decisions are mostly in the configuration.
Why poll every minute? Because CloudWatch alarms evaluate on metric periods, and a 1 minute period gives us a 1 minute reaction time to a full queue. Our instances take about a minute and a half to launch and register as runners, so the total time from “all slots full” to “new runner available” is around 2-3 minutes. That’s fast enough that most queued jobs only wait one cycle.
Why a scale-out threshold of 4? Because we run 4 runner processes per instance. When all 4 are busy, the next job to arrive has nowhere to go. Setting the threshold at 4 means we trigger scale-out exactly when the queue would start building. Setting it lower would mean scaling out while there are still free slots, which wastes money. Setting it higher isn’t possible — you can’t have more than 4 busy runners on a 4-runner instance.
Why scale-in at 0 for 5 minutes? Because we don’t want the extra instance oscillating up and down between bursts. If a developer pushes 3 PRs in sequence, we don’t want to scale in after the first batch finishes and then immediately scale out again when the next batch starts. The 5 minute window with a 300 second cooldown gives enough buffer for back-to-back pushes without keeping the instance running for hours after everyone’s gone home.
Why SimpleScaling instead of StepScaling? Honestly, because StepScaling doesn’t support cooldown periods and I only discovered that when the Terraform apply failed halfway through. SimpleScaling does exactly what we need — add one instance when busy, remove one when idle. There’s no need for graduated steps when the ASG is scaling between 1 and 2 instances.
What about treat_missing_data? Both alarms use notBreaching. If the Lambda fails to publish a metric for a few minutes — maybe the GitHub API is slow, maybe the Lambda times out — we don’t want that to trigger a scale-in. Missing data means “we don’t know,” not “nobody’s busy.” This is a small detail that prevents a bad scenario where a Lambda failure causes the ASG to scale down and drop running jobs.
I tested this in dev first by opening 5 draft PRs at once against the robot software repo. Each PR triggers an integration workflow that uses the self-hosted runners, so 5 PRs means 5+ jobs competing for 4 runner slots.
The first dev test showed the autoscaling working but exposed the bootstrap latency. The scale-out alarm fired within about 15 seconds of all slots filling up, and the ASG launched a new instance almost immediately. But the new instance took about 5 minutes from launch to picking up its first job — the runner binary had to download, register with GitHub, and start the listener processes. That first-job latency was too slow for what I wanted.
The AMI baking work from Part 2 is what fixed this. With the runner binary and dependencies pre-baked into the AMI, the bootstrap got much faster. By the time I validated in production, the scale-out alarm was firing about 1 minute 42 seconds after the burst started, and the new instance was launching 8 seconds later. Total time from burst to new capacity was under 2 minutes.
The scale-in worked cleanly too. After all the PR jobs finished, the busy runner count dropped to 0, the alarm waited through its 5 evaluation periods, and the ASG scaled back to 1. No manual cleanup, no orphaned instances.
The production autoscaling has been running since mid-March 2026. In the first few days of real usage, the extra instance ran for about 2.5 hours total during burst periods. That’s roughly 60 cents.
For a system that eliminates queue waits during peak development hours and requires zero manual intervention, that’s a pretty good trade. The Lambda itself runs within the free tier — 128MB of memory, 30 second timeout, once per minute. The CloudWatch metrics and alarms add negligible cost.
The most expensive part of the whole setup was the time I spent building the queue-wait report to justify it.
For anyone thinking about doing something similar, here’s what the infrastructure looks like:
One Lambda function (Python 3.12, 128MB, 30s timeout) triggered every minute by EventBridge. It reads a GitHub PAT from SSM, paginates the org runners API, counts busy runners by label, and publishes a CloudWatch custom metric.
Two CloudWatch alarms watching that metric — one for scale-out (busy count >= runner concurrency), one for scale-in (busy count = 0 for N consecutive periods). Both use the Maximum statistic and treat missing data as not breaching.
Two ASG SimpleScaling policies — one that adds 1 instance, one that removes 1. Cooldowns of 60 seconds and 300 seconds respectively.
The whole thing is behind an enable_job_autoscaling feature flag in Terraform, so it can be toggled per environment. Dev and prod each have their own Lambda, alarms, and policies, publishing metrics with different runner label dimensions so they don’t interfere with each other.
The autoscaling is running and the cost is negligible, but there’s a bigger story here about how the tooling around it evolved. The Lambda handler was one of the first things I wrote in Python for this project. The rest of the infrastructure — the runner bootstrap, the AMI baking, the operational CLI — started as bash and stayed as bash longer than it should have. That migration, and why it mattered, is Part 4.
This is Part 3 of a series on building production-grade self-hosted GitHub Actions runners at a robotics startup.
What I didn’t talk about was how we build that AMI. And honestly, this is where things got interesting.
We don’t use Packer. We don’t use EC2 Image Builder. We don’t have a separate build server or a dedicated image pipeline. Instead, we bake our AMIs in-place — we take the running dev instance, patch it, update the cached Docker images, snapshot it, and roll forward. The whole thing is orchestrated by a single GitHub Actions workflow.
It works really well now. Getting here was a different story.
Packer is the standard answer to “how do I build AMIs” and for good reason — it’s mature, well-documented, and handles a lot of the complexity for you. But when I looked at what we actually needed, Packer felt like it was solving a bigger problem than we had.
Our setup is a single EC2 instance running GitHub Actions runners. The AMI doesn’t need to be built from scratch each time — it needs to be updated. New OS patches, a fresh pull of the devcontainer image, maybe a new version of the runner binary. The base doesn’t change. The delta is small.
Building from scratch with Packer would mean spinning up a temporary instance, running a full provisioning script, waiting for a 23GB Docker image to download over the network, creating the AMI, and tearing it all down. That’s 30+ minutes of build time and a bunch of infrastructure to maintain for something we only do once a week.
What if instead we just took the instance that’s already running — the one that already has the 23GB image cached, the runner binary installed, all the dependencies in place — and turned that into the next AMI?
Here’s what the pipeline actually does, step by step:
1. Detach the instance from the ASG. We lower the ASG min_size to 0 and detach the current instance. This is important because we need to stop the instance to create a clean snapshot, but we don’t want the ASG to panic and launch a replacement while we’re working.
2. Patch via SSM. Once the instance is detached, we run AWS-RunPatchBaseline through Systems Manager. This applies any pending OS security patches without needing SSH access. The instance is still running at this point, we’re just using it as a build target.
3. Pre-pull the latest devcontainer image. This is the step that matters most for our use case. We send an SSM command that logs into GHCR, pulls the latest devcontainer:stable image, and verifies it landed correctly. Because the instance already has the previous version cached, Docker only downloads the changed layers. What would be a 23GB pull from scratch is usually a few hundred MB of deltas.
4. Clean up runner state. Before snapshotting, we need to remove anything that’s specific to this particular instance — runner registration credentials, systemd service state, work directories, cloud-init caches. If any of this ends up in the AMI, the next instance that boots from it will have identity conflicts.
5. Stop the instance and create the AMI. We stop the instance (not terminate), create an image from it, tag the AMI and its snapshots, and wait for it to become available.
6. Update the SSM parameter and roll forward. The new AMI ID gets written to an SSM parameter. Then we run terraform plan and terraform apply, which picks up the new AMI ID and updates the launch template. The ASG launches a fresh instance from the new AMI.
7. Smoke test. We wait for the new instance to come up, verify via SSM that the runner service is healthy, and only then terminate the old detached instance.
The whole process takes about 15-20 minutes, most of which is waiting for the AMI to finish creating and the new instance to boot. The actual “work” — patching, pulling, cleaning — is maybe 5 minutes.
The first version of this pipeline worked on the happy path. The problem was that AMI operations on AWS have a lot of unhappy paths, and I found most of them on November 6th, 2025.
I was adding AMI cleanup — the part of the pipeline that deletes old AMIs so they don’t pile up and cost money. Simple enough in theory: list all AMIs matching a naming pattern, keep the newest ones, delete the rest along with their EBS snapshots.
The first version used inline bash in the GitHub Actions workflow, shelling out to the AWS CLI and piping through jq for JSON parsing. It worked locally. It did not work in CI.
Here’s what went wrong, roughly in the order I discovered it:
Two machines, two different toolsets. The GitHub-hosted runner where the orchestration runs and the EC2 instance where the commands actually execute are two completely different environments with different tools installed. What’s available on one isn’t necessarily available on the other, and I was making assumptions about both.
The AWS CLI --query parameter is unforgiving. I had a filter that was supposed to exclude the newest AMI from the deletion list, but the sort order wasn’t doing what I thought it was. The cleanup was keeping the wrong AMI and trying to delete the one it should have kept. These are the kinds of bugs you don’t catch until you have enough data to exercise the edge case — in this case, more than 2 AMIs.
Eventually-consistent API responses. Sometimes describe-images returns an empty list briefly after a new AMI is created but before it’s fully registered. My code assumed the list would always have at least one entry and crashed with an index error. Needed retry logic for empty state responses.
Bash arithmetic will betray you. I had a timeout calculation where operator precedence silently changed the meaning of the expression. The timeout was effectively infinite for the first few iterations and then immediately expired.
Snapshot lifecycle quirks. When you deregister an AMI, its EBS snapshots don’t automatically get deleted — you have to delete them separately. But sometimes the snapshot is briefly “in use” even after the AMI is deregistered, and the delete call fails. More retry logic, more error handling.
Inconsistent JSON output. The AWS CLI’s --output json flag formats things differently depending on whether there’s one result or many. A single AMI ID comes back as a string, multiple come back as an array. The bash script was handling one case but not the other.
That was 22 commits in a single day. Every time I’d fix one issue, the pipeline would get a little further and hit the next one. By the end of the day I had a working cleanup process, but the bash was getting messy. I knew I’d need to rewrite it properly at some point, but I had other priorities and it was working — I’ll talk more about that migration in Part 4 of this series.
One of the better decisions I made during that November 6th marathon was adding a safety cap to AMI deletions. The cleanup takes a max_delete_per_run parameter, and if the number of AMIs to delete exceeds that limit, it refuses to run and throws an error instead.
I started with a cap of 3. Within a few weeks, normal operation pushed us past that — we had a stretch where the weekly bake ran but cleanup kept failing silently, so AMIs accumulated. Bumped it to 5 in December. Hit the limit again in January after another cleanup regression. Bumped it to 10.
The cap seems conservative, but it’s there to prevent a bug in the AMI listing logic from accidentally deleting everything. If some API response comes back wrong and the “keep” list is empty, the cap ensures we delete at most N images instead of all of them. It’s a safety net for exactly the kind of API edge cases that caused all those November problems.
The logic itself is simple: the AMI list comes in sorted by creation date, we keep the newest N, and we refuse to delete more than the safety cap. That’s it. The simplicity is the point — when something guards against accidental deletion, you want it to be easy to reason about.
The bake pipeline only touches the dev environment. It creates the AMI from the dev instance, writes the new AMI ID to an SSM parameter, and rolls the dev ASG forward. Prod is a completely separate concern.
Getting the new AMI into production used to be a manual step — go to the GitHub Actions UI, pick the promote_ami action, trigger it. That worked fine, but it meant someone had to remember to do it. Now the whole thing is automated as part of the weekly scheduled run.
Every Sunday, the pipeline runs end-to-end: bake the dev AMI, promote the AMI ID from the dev parameter to the prod parameter, run terraform apply against prod to pick up the new launch template, roll any outdated runner instances, and send a Slack notification with the result. If any step fails, the pipeline stops and Slack gets a failure notification instead. The manual promote is still there in the prod workflow as a fallback if something needs to jump the queue.
The promotion itself is deliberately boring — it reads one SSM parameter and writes the value to another. But the pipeline around it means dev always bakes first, and if the bake fails, prod doesn’t get touched. Running in dev for a week before promoting to prod has caught real problems that the smoke test didn’t.
This post covered the AMI pipeline — how we build it, what broke along the way, and how we promote changes safely from dev to prod. There’s more to say about the bash-to-Python migration specifically: the runner bootstrap went through the same journey, and that rewrite touched a lot more than just the AMI cleanup code. I’ll cover that properly in Part 4.
But first — there’s a question I’ve been dodging: how does the system know when to scale? We don’t use webhooks. We don’t use GitHub’s built-in autoscaling. We have a Lambda function that runs every minute, checks how many runners are busy, and publishes a CloudWatch metric. The ASG scales off that metric. It’s about 170 lines of code and it’s been more reliable than any webhook-based approach I’ve seen.
This is Part 2 of a series on building production-grade self-hosted GitHub Actions runners at a robotics startup.
7 months and 166 commits later, we’re running our entire CI pipeline — builds, tests, linting, Docker-based jobs, the lot — on a single EC2 instance. One. Not a fleet. Not Kubernetes. Not some managed runner service that charges per minute. One m7i.xlarge in eu-central-1.
And it handles everything.
Here’s the thing nobody warns you about when you’re building robotics software with GitHub Actions: your devcontainer image will be enormous, and GitHub really does not want you to use it.
Our devcontainer image is about 23GB. It has ROS 2 Humble, a full robotics toolchain, simulation dependencies, OpenCV, BLAS libraries, colcon build tools — everything you need to build and test a robotics monorepo. It’s big because robotics is big. There’s no way around it.
Standard GitHub-hosted runners have 14GB of disk space. Our image literally does not fit. So right from the start, we were forced onto GitHub’s Large-GPU-Runner — a beefier hosted runner that costs significantly more per minute. And even on the large runner, every single CI run started by pulling that 23GB image from scratch. That pull alone took 6-8 minutes. Every. Single. Run.
We looked at caching it. GitHub’s built-in action cache has a 10GB limit. The Docker cache action is backed by the same 10GB store. Our image is 23GB. Dead end.
So before a single line of our code even started compiling, we were already 6-8 minutes into the job and burning money on oversized hosted runners. Multiply that by every PR, every push, every developer on the team. At a startup, time and money are the same thing, and we were wasting both.
Self-hosted runners were the obvious fix. But the question was how.
I looked at the options. There are managed self-hosted runner services. There’s the actions-runner-controller for Kubernetes. There are Terraform modules with hundreds of variables that try to cover every possible use case.
They all felt like overkill for what we needed.
We’re a small team. We don’t have a Kubernetes cluster to throw runners onto (and honestly, running K8s just to host CI runners is a special kind of infrastructure madness). The managed services charge per-minute pricing that, once you do the maths, isn’t that different from just paying GitHub directly. And the big open-source Terraform modules were so configurable that configuring them felt like a project in itself.
So I did what any engineer does when the existing tools don’t fit. I opened a new repo, wrote an initial commit, and started building.
Here’s the entire infrastructure:
m7i.xlarge EC2 instance (4 vCPUs, 16 GiB RAM)That’s it. No load balancer. No container orchestration. No service mesh. The instance boots, registers itself with GitHub, and starts picking up jobs.
The key insight — and this took me a while to land on — is runner concurrency. A single GitHub Actions runner process can only handle one job at a time. But nothing stops you from running multiple runner processes on the same machine. So I run 4. One per vCPU.
When the instance launches, 4 separate runner processes register with our GitHub org. GitHub sees 4 idle runners, all with the same labels, and distributes jobs across them automatically. 4 concurrent CI jobs, one instance. Each runner gets about 1 vCPU and 4 GiB RAM, which is plenty for our workloads.
The concurrency trick is nice, but the real win is what’s baked into the AMI.
Remember that 6-8 minute image pull? Gone. Completely gone. Our custom AMI has the 23GB devcontainer image pre-pulled into the Docker cache. When a CI job starts and specifies container: image: ghcr.io/...devcontainer:stable, Docker finds it locally and starts it in seconds. No network transfer, no waiting, no praying that GHCR doesn’t throttle you.
We bake a fresh AMI weekly. The pipeline detaches the current instance from the ASG, patches it via SSM, pre-pulls the latest devcontainer image, snapshots it, rolls the ASG forward to the new AMI, runs a smoke test, and terminates the old instance. No Packer, no separate build infrastructure — just GitHub Actions orchestrating AWS APIs.
That AMI baking pipeline deserves its own post (the number of edge cases we hit building it was genuinely absurd), but the point is: eliminating the image pull is what made the biggest single difference. Everything else is optimisation on top.
On the very first day I set this up, I actually ran a side-by-side comparison — the same test suite running on both the Large-GPU-Runner and the new self-hosted runner in parallel. The difference was immediately obvious. All that time we’d been sinking into image pulls, environment setup, and cold caches just vanished.
My first version used ephemeral runners. Fresh registration for every job, clean state every time. It’s what most guides recommend and it feels “correct” from a security standpoint.
It was also slow.
Every job had to re-register the runner with GitHub, pull a fresh token, configure the runner binary. That’s 30-60 seconds of overhead per job, and when you’re running dozens of jobs a day across 4 runner slots, it adds up. Worse, you lose all your workspace caches between jobs. Git clones start from scratch, dependency caches are gone, build artifacts disappear.
About two months in, I switched to persistent runners. The runner processes start when the instance boots and they stay up. They pick up jobs, run them, pick up the next one. State persists between runs, which means git fetches are incremental instead of full clones, Docker layers stay warm, and builds that benefit from cached outputs get those benefits.
The tradeoff is that you need to actively manage workspace hygiene. Stale git lock files from killed jobs, leftover LFS hooks from previous checkouts, repo-local git config that bleeds between projects — all real problems we hit. I ended up writing a pre-job hook that runs before every single job and cleans all of this up. It was annoying to build, but it runs in under a second and it solved the reliability issues completely.
That’s the thing about self-hosted runners that nobody writes about. The initial setup takes a day. Getting it production-ready takes months. The edge cases that only surface when you’ve been running hundreds of jobs a week — those are where the real work is. But once you’ve solved them, they stay solved.
This is the part everyone actually cares about, so let me be real about it.
An m7i.xlarge on-demand in eu-central-1 runs about $170/month. We could cut that significantly with reserved instances or savings plans, but we haven’t bothered yet because even at on-demand pricing, the maths works out easily.
Compare that to what we were paying before. The Large-GPU-Runner is one of GitHub’s premium hosted runner tiers — you’re paying a meaningful premium per minute for those. And we had no choice but to use them because standard runners couldn’t even fit our image. With self-hosted, we went from paying per-minute for oversized hosted runners to a flat monthly cost for a machine that’s always warm and always ready.
But the real cost saving isn’t the compute bill. It’s developer time. When every CI run was burning 6-8 minutes just pulling a container image before doing anything useful, that’s dead time on every single push. Across a team, across a day, across a sprint — it compounds into something massive. Eliminating that wait was worth more than any line item on the AWS bill.
For the first 6 months, we just ran a single instance with max_size = 1. No autoscaling. One box, 4 runners. It handled everything.
Recently, as the team and the number of pipelines grew, we started occasionally saturating all 4 slots. So I added job-driven autoscaling: a Lambda function runs every minute, calls the GitHub API to count how many runners with our label are currently busy, and publishes that as a CloudWatch metric. Standard ASG scaling policies react to it — scale out by 1 when all 4 runners are busy, scale back in when they’ve been idle for 5 minutes.
The whole autoscaling layer is about 170 lines of Python and some Terraform resources. It costs almost nothing to run (a Lambda invocation per minute is well within free tier), and it means we can burst to 8 concurrent jobs when we need to without paying for that second instance when we don’t.
I deliberately chose this polling approach over webhook-based autoscaling. Webhooks mean exposing a public endpoint, managing authentication, handling delivery failures. The polling model is simpler, cheaper, and “good enough” at 60-second granularity. We also did a bunch of work to get the ASG instance startup time down — started at around 5 minutes, got it to just over a minute. I’ll dig into the detail of that in a later post in this series.
If I started this project over, I’d skip ephemeral runners entirely and go straight to persistent. The security argument for ephemeral runners makes sense in theory, but in practice, if you’re running your own infrastructure, you’re already trusting the machine. The performance difference is too significant to ignore.
I’d also write the bootstrap runtime in Python from day one instead of starting with shell scripts. The first version was a cloud-init template with bash doing all the heavy lifting — fetching tokens, registering runners, managing systemd services. It worked, but it was brittle and nearly impossible to test. The Python rewrite gave us proper error handling, structured logging, and actual unit tests. Should have done that from the start.
Everything else — the Terraform structure, the ASG-based approach, the concurrency model — I’d keep exactly the same. It’s been running in production for 7 months, it’s handled thousands of CI jobs, and the total infrastructure cost is less than what most teams spend on a single SaaS tool.
You don’t need a complex orchestration layer to run self-hosted GitHub Actions runners. You don’t need Kubernetes. You don’t need a managed service. For most small-to-medium teams, a single well-configured EC2 instance with multiple concurrent runner processes will handle more than you think.
The trick is getting the foundations right: a custom AMI with your dependencies baked in, persistent runners for fast job starts, good workspace hygiene between jobs, and just enough autoscaling to handle burst load.
Start simple. You can always add complexity later. But you probably won’t need to as soon as you think.
This is Part 1 of a series on building production-grade self-hosted GitHub Actions runners. Coming up next: