u/Beginning_Brush_6770

Hey gophers,

I’m building an evaluation engine (essentially an online judge) that compiles and simulates user-submitted hardware design code. The worker pool is written in Go, and it spins up ephemeral Docker containers via the Docker Go SDK to sandbox the execution environment and run synthesis tools.

The challenge is safely evaluating untrusted code that might contain infinite loops, fork bombs, or massive memory allocations, while ensuring the Go worker pool stays resilient and doesn't leak resources.

My current approach:

1. Receive job from the queue (Redis/RabbitMQ).
2. Spin up a container with strict CPU and memory limits applied via HostConfig.
3. Drop all network privileges (NetworkMode: "none") and mount a read-only filesystem, except for a temporary volume for the specific job.
4. Wrap the execution in a context.WithTimeout (e.g., 5 seconds) passed to ContainerWait and ContainerLogs.
5. If the context deadline exceeds, aggressively call ContainerKill and clean up.

My concerns where I'd love some battle-tested advice:

• Zombie Containers: If a Go worker panics or the host machine hard-crashes mid-execution before the deferred ContainerRemove fires, orphaned containers are left spinning. Do you typically handle this with a separate reaper service/cron job, or is there a cleaner pattern within Go?
• Context Cancellation Edge Cases: Are there known pitfalls in the Docker Go SDK where streaming logs (ContainerLogs) from a hijacked container could block a goroutine indefinitely, even when the parent context is cancelled?
• The Next Level: At what scale or security threshold did you find Docker insufficient and migrate to raw containerd, runc, or microVMs like Firecracker?

Would love to hear war stories from anyone who has built CI runners, sandboxes, or remote code execution platforms in Go.

Thanks!