Our team chat runs on our own Matrix server, with local LLM agents in the rooms. Wrote up the setup.
Between the EU Chat Act and not wanting to hand my data to LLM vendors, I moved my team’s chat, our documents and the agents’ memory onto a local machine we run ourselves. We’ve been running it for about half a year now and couldn’t be happier, no issues or hiccups, so I wanted to write it up in case it helps someone doing the same.
We run Synapse, and the agents are just Matrix users in our rooms. Mention one and it replies, usually from a local model. Nothing’s exposed to the internet. Phones and laptops get in over Tailscale.
Every agent goes through a tool gateway. The rule is that an agent on a hosted model never gets a tool that reads local data. The gateway refuses to start if the config allows it, and checks again on every call. It also caps each agent’s tokens per day, which you’ll want once a hosted model is in the mix. Hosted models are opt-in, and one only sees the rooms you invite it to.
There’s also a scribe running in the background on a local model. Once a day it reads the rooms and files the day into a local Qdrant index, so agents can search old conversations without anything leaving the machine. It indexes the links people share but doesn’t fetch them unless you turn that on.
The repo is docs, specs and compose templates. We run our own builds of the gateway and the scribe, but the repo has them as specs only, so you can build your own or swap in something that exists. It’s Linux first. Only the homeserver chapter has been tested on a clean machine, and there’s nothing on backups or hardening. The README also lists the metadata that leaves anyway: device info on the overlay, certificate transparency logs, and phone notifications, which go through Apple’s and Google’s push services (they see that a message arrived, but not what it says).