Alex Edwards

I’m starting work on a remote AI agent harness for my homelab, specifically to complement the tools I already use by giving them somewhere to delegate parallel or long-running work.

As more tools add agentic integrations, I’m interested in the infrastructure between them: the systems that allow agents to delegate work safely elsewhere.

I’m used to sharing my workspace with an agent. This can mean repeatedly approving its actions or giving it permissions I would not want it to have outside a sandbox.

Why now

I’ve recently completed some much-needed maintenance in my homelab. Alongside updating the software, I finally created a Proxmox cluster and ported some critical services over to a Raspberry Pi 5 that had been sitting idle. I’ve also been relying more on a cloud instance of Arch Linux as a development workspace that I can reach over SSH from anywhere.

I’ve also revitalised my observability warehouse and started hosting an OpenTelemetry collector. This gives me more insight into both the homelab and the AI harnesses I use, including OpenCode personally and GitHub Copilot at work.

I’m drawing inspiration from services such as Amazon Bedrock AgentCore and Cloudflare’s Durable Objects to build an agent harness service with authentication and a filesystem for performing operations. The first outcome I want is an API for creating and managing ephemeral workspaces. Like the other services I self-host, it’s equal parts building something from scratch, experimenting, and figuring out how things work.

The hard part

Running agents remotely creates a different set of problems. Rather than giving an environment the same permissions I have, I need to start from zero trust. Policies must be scoped and secrets must be revocable. A human should be the escalation path for exceptional decisions, not someone who monitors every command.

First proof of concept

The first proof of concept will test whether I can create an isolated runtime, install the agent and its dependencies, run a task, and then destroy the runtime. Before building that orchestration myself, I’ll investigate which parts existing open-source projects already provide.

Architecture direction

My initial thinking is to borrow from Kubernetes, with a control plane managing a set of worker nodes.

When a request arrives, the control plane would create a session and associate it with its policies and capabilities. It would then place the session on a worker VM, which would create and manage an isolated microVM for the agent.

A gateway API will expose session creation and status, relay communication between users or agents and the remote harness, and receive escalation callbacks. The control plane will manage the underlying workers and session state.

I don’t know what the final system will look like yet. The first step is to work out how to create, customise and destroy one isolated runtime reliably. That should tell me what the wider architecture needs to be.