u/Creative_Factor8633

Container → microVM is not the finish line. Your isolation boundary is not in the Guest kernel. It's in that root process on your host called virtiofsd.

1. Everyone just moved house

For the past six months, every vendor still serious about agent sandboxes has been telling the same story:

Shared kernels are over. We've upgraded to Firecracker / Kata / Cloud Hypervisor. Each tenant gets its own Guest kernel = hardware-level isolation = safe.*

That story is more honest than the shared-kernel one. That's it.

E2B prints "Firecracker" on the homepage. Modal blogs about gVisor. Kata is the silver bullet of the K8s crowd. 90ms cold start, written in Rust, 5 MiB memory overhead. Sounds airtight.

Until you ps aux | grep -E '(virtiofsd|vhost)' on the host.

2. virtiofsd: the root daemon sitting next door

To let the Guest reach host volumes at near-native speed, the standard microVM stack runs a daemon on the host called virtiofsd, wired to the Guest over the virtio-fs channel. What permissions does it have?

Host root.

Not a misconfiguration — by design. It has to act on the host filesystem on the Guest's behalf.

USENIX Security '23 gave this an unflattering name: Operation Forwarding Attacks.

Some Guest syscalls get forwarded to that high-privileged proxy on the host for execution. Physical isolation? Sidestepped.

CVE-2022-0358 walked it through end-to-end: a plain open() from inside the container is forwarded across virtio to virtiofsd, which then bypasses the host's inode_init_owner() check and writes a file with root SGID into a shared host directory.

Container root → host root. The hardware boundary of the MicroVM was never crossed. It was flanked.

3. It's not just virtiofsd

Forwarding surface Attack shape Measured impact
virtiofsd (file) Daemon privilege abuse Container → host root (CVE-2022-0358)
virtio-blk (storage) I/O amplification Co-located neighbor I/O drops 93.4%
virtio-net (network) Packet-parse amplification Host kernel nf_conntrack table fills instantly
vhost-net / KVM PIT worker threads cgroup attribution missing Guest borrows host kernel-thread cycles, bypasses vCPU quota

Same shape every row: the physical boundary is fine, the operation-forwarding pipes either side of it are not.

Each pipe has a host-side proxy: a daemon, the VMM main process, a host kernel thread. Each proxy is more privileged than anything in the Guest. All the Guest needs is to make the proxy do something on its behalf — and now it speaks with the proxy's voice.

Upgrading to MicroVM doesn't make these proxies disappear. It moves them from "kernel namespace bookkeeping" to "a row of root daemons in host userspace." The attack surface didn't vanish. It moved.

4. The industry answer is "nest one more layer"

  • vhost-user offload: peel virtual devices out of the VMM main process, run them as isolated low-privilege daemons.
  • Reverse user namespace: use a user namespace to strip virtiofsd of real host root before letting it serve the Guest.
  • Jailer: lock the VMM into chroot + cgroups + tight seccomp (Firecracker's Jailer allows just 24 syscalls and 30 ioctls).
  • Matryoshka: bare metal → Jailer-wrapped VMM → ephemeral Guest kernel → OCI container inside Guest → agent code inside container. Every layer distrusts the next.

This works. The cost: you now have N more long-lived host daemons to audit, patch, and authorize. Every nesting layer adds another permanent privileged process to the host inventory.

So i guess we need a different way for the agent run in the sandbox. What proposal do you have?

reddit.com
u/Creative_Factor8633 — 17 days ago

I run engineering on a small embedded-sandbox project. A handful of news items dropped recently — an a16z agent escape post-mortem, a CVE on an open-source agent gateway (ClawBleed, ~42k instances exposed), Cloudflare's new Outbound Workers for Sandboxes, and NIST's agent-security concept paper. They look unrelated. They're not.

Every one of them is patching the same architectural problem: the agent sandbox lives in someone else's datacenter, and its security boundary is whoever it happens to be talking to. I've started calling it the "front desk problem" — there's always a control plane, the agent can reach it, and "reachable from inside" is the entire game.

Concrete: in the a16z escape, the setup was solid. Anvil forked from mainnet, Docker firewall dropping all egress, no privileged tools. The agent ran:

cast rpc anvil_nodeInfo

…which dutifully printed the upstream RPC URL and the Alchemy API key in plaintext. Pivot, anvil_reset, fast-forward, mine its own blocks, working PoC. This isn't a CVE. Nothing was exploited in the traditional sense. The agent just noticed it was living in someone else's apartment and the master key was sitting at the front desk in a metadata endpoint. It walked over and asked.

Lining the four up:

  • a16z escape: sandbox shares a control plane with the thing it's trying to contain.
  • ClawBleed: a gateway process trusted by default by anything on the same machine.
  • Cloudflare Outbound Workers: token proxy outside the box, because the inside can't be trusted to hold its own credentials.
  • NIST + GKE Agent Identity: stamping every agent with a cryptographic ID, because at the platform layer you genuinely cannot tell which agent pulled which trigger.

All rational responses. To a paradigm I've quietly stopped believing in.

I don't think the cloud-sandbox category goes away. Multi-tenant SaaS that runs strangers' code, GPU passthrough, geo distribution — that's their corner. But a non-trivial slice of agent workloads — anything privacy-sensitive, high tool-call frequency, or offline — is better served by a sandbox that boots inside the agent's own process: no daemon, no socket, no RPC control plane, security boundary at the local hypervisor (KVM on Linux, Hypervisor.framework on macOS). No front desk to walk up to.

Honest tradeoffs of going local: cold start is 100–500ms not sub-ms; GPU passthrough is rough (Modal still wins fine-tuning); no autoscaling.

What I'm least sure about: whether cold-start on the cloud side closes fast enough that the network-hop argument stops mattering for tight agent loops. Curious what folks here are seeing on tool-call latency lately.

BTW: I work on BoxLite, an embedded MicroVM sandbox in this space. Putting GitHub link in the comments

reddit.com
u/Creative_Factor8633 — 22 days ago

When OpenAI shipped the Agents SDK, they listed 7 official cloud sandbox providers. We tried them. They work. But if you're doing anything beyond a quick prototype, the economics and the latency break down fast.

Here's the reality of running agents on hosted sandboxes:

  1. Network Latency kills you: An average agent task makes 8-15 tool calls. On a hosted sandbox, every exec_command requires two internet round trips. You aren't paying for the 90ms cold start; you're paying for those 30 network hops.
  2. State management is a nightmare: Cloud sandboxes are ephemeral sessions. You can't just pause an agent, leave it for 12 hours, and resume instantly. You end up reinstalling pandas and torch every single turn.
  3. Swarm economics: If you want to spin up 1,000 agents for an evaluation, per-second billing on 1,000 cloud sandboxes is brutal.

The default UnixLocalSandboxClient in the SDK (bubblewrap on Linux, seatbelt on macOS) runs locally, but it shares your host kernel. If you're running LLM-generated code, a syscall filter isn't a real security boundary.

The Solution: Hardware Virtualization, Locally

We built BoxLite to solve this. Think of it as the SQLite of sandboxing.

It gives the agent a dedicated guest kernel running inside KVM (Linux) or Hypervisor.framework (macOS). The boundary is a hardware virtualization fault, not a process filter.

  • No Daemon. No Socket. No Root. It's a single pip install boxlite-openai-agents.
  • Zero Network Hops: The microVM runs in the same process as your agent runtime. IPC is microsecond-level.
  • QCOW2 Snapshots: We use native QCOW2 copy-on-write. You install a heavy dependency once, snapshot the workspace, and you can resume it (or fork it 1,000 times for a swarm) in sub-100ms.

If you are building coding agents on private repos, dealing with personal data, or need to run 100% air-gapped with a local vLLM, you can't rely on a cloud sandbox.

You can swap out the default client with one line based on your OpenAI Agent SDK:

client = BoxLiteSandboxClient()

If you've been fighting cloud sandbox latency or security compliance, check out the implementation.

It's open source. Happy to answer any questions about the KVM/HVF architecture below.

u/Creative_Factor8633 — 25 days ago