I built the smart speaker we always wanted
I wanted to see if Claude can handle Vibe Hardware Engineering to help me make a smart speaker. Turns out, it can! I call it boxBot.
It helped select the hardware set, raspberry pi, Hailo , respeaker mic, pi camera, waveshare screen and speakers. Helped me calculate thermal loads and dissipation rates for a passive cooling setup. I made the box by hand out of walnut.
The agent inside is custom as well. You could probably throw openclaw on it and call it a day but I wanted to craft something that was tightly coupled with the hardware more secured considering it’s sitting in my living room with a camera and mic. The agent is highly skills driven with only a small set of tools, everything else goes through Python scripts and a custom made boxBot sdk the agent can use to control the box and the display.
The display system uses a widget framework so the agent can easily read what’s displayed without a screenshot and can effectively manipulate what’s on the screen. The agent uses json to specify how the widgets should be arranged on the screen and what data should flow into them.
When building a smart speaker, there’s a lot of nuance to human conversation that voice agents really struggle with, like background noise, side conversations, barge-in, etc. I was able to simplify the logic a ton by making it agent driven, the agent can control when to mute the mic to ignore background chatter, it decides what order to work vs talk, it can choose what channel to respond in; voice or WhatsApp. Instead of complex rules, agent driven hardware plus skills can provide a much richer experience, now that boxBot manages the family calendar my wife wants a text whenever I put something on it, boxBot updated the calendar skill with that request so now when I add something, it sends her a message. Just one line in a .md file and you get the desired behavior. It’s incredibly flexible and simple.
I could nerd out on the details about the memory system, struggles with woodworking, and security details but I’ll save that for the comments if people want to chat.
It’s open sourced if you want to inspect. Still a work in progress but after a few months it is finally feeling like a useful assistant to the family day-to-day.
Www.github.com/dv-hart/boxbot