A while back I posted about observing my Home Assistant setup, and mentioned I'd added native Prometheus instrumentation to Zigbee2MQTT. Those PRs are still open upstream, and a few people have expressed interest in trying it themselves, so I've packaged the branches up as an add-on repository you can just install: https://github.com/tomwilkie/zigbee2mqtt-prometheus

Add it the usual way - under Settings > Add-ons > Add-on store > Repositories, install "Zigbee2MQTT (Prometheus)", and you get metrics on port 9142. It's built from Zigbee2MQTT 2.13.0, and I'll do my best to build new versions every time theres an update upstream.

Grafana dashboards for it are here, an overview with per device drilldown plus a coordinator view:

https://preview.redd.it/5tlfe2t1f0hh1.png?width=2628&format=png&auto=webp&s=75a07ea327cee7066fa2ed8f60b757c883fba79c

All of this goes away then the PRs land. Until then the repo (warning: lots of claude) also has the runbook for rebasing the forks onto each new Zigbee2MQTT release, so it can keep up. PRs, if you want to nudge them along ;-)

• https://github.com/Koenkk/zigbee2mqtt/pull/31645
• https://github.com/Koenkk/zigbee-herdsman/pull/1751

I did a big write up of my observability setup for Home Assistant, check it out: https://www.reddit.com/r/homeassistant/comments/1uinw9t/observing_home_assistant/

I recently had a couple of months of parental leave and I wanted a project I could do, at home, without it being too much like “work”. So I set about getting a really comprehensive observability setup for my Home Assistant stack, and in the process learnt a ton about HAOS, Home Assistant, ZigBee and more.

Disclaimers: I work at Grafana Labs. I used a lot of Claude Code to build this, but this post is not about AI and was not written by AI! Everything in the post can be done with the open source Grafana, Alloy, Prometheus and Loki.

“Layer 0”: The Hardware and Operating System

First I wanted to understand how “busy” my Home Assistant box was - I was using a decade-plus-old Intel NUC, and was noticing some lagginess in the UI.

We do publish a Home Assistant app to gather telemetry but it didn’t have the standard Prometheus node-exporter enabled. I added this but quickly realised the values it reported were plain wrong. You see, HAOS apps run inside the Supervisor's managed Docker environment, which does not permit mounting the host's /proc or /sys filesystems into a container. Those mounts are required for accurate node exporter metrics (CPU, memory, disk, network) at the host level. This is a known, deliberate restriction - see:

• community.home-assistant.io — Mount host's /proc & /sys into add-on container
• github.com/orgs/home-assistant — Discussion #3203

To get around this I deployed Alloy (which embeds node-exporter and the Prometheus agent) in a plain old docker container, and by using DOCKER_HOST=ssh://root@homeassistant.local I was able to version control the docker compose setup and use a GitOps-style deployment “pipeline” from my laptop. The downside of this approach is the weekly “your system is unsupported” messages you get in Home Assistant…

With this in place, gathering host-level metrics and logs, I was able to pretty quickly see my machine’s CPU was quite busy, perhaps explaining the sluggishness I was experiencing. I upgraded to a newer, faster mini PC (a GMKtec M7 Ultra) but hit a new snag - it came with a cheap SSD that failed within days. Having the journal (i.e. kernel logs) stored off-machine meant I could quickly diagnose what went wrong and get a new (expensive, Samsung) SSD same-day delivered.

Finally, I also took a look at the temperature of things like the CPU - and found there was far too much heat in my server rack. I whacked a big Noctua fan I had left over from my 3D-printed wind tunnel project on top of the rack to pull out the heat and saw an immediate difference.

\"Standard\" Linux Dashboard

“Layer 1”: Docker (for apps)

I run a ton of HA apps - Zigbee2MQTT, Z-Wave JS, VSCode, ESPHome Builder, Music Assistant etc. I wanted to see how they were contributing to system load, and start debugging why tracks were skipping with Music Assistant. For that I needed per-container system telemetry and logs. Typically you’d use cAdvisor to translate the metrics into Prometheus format and luckily Alloy embeds that too. Again it needs to mount parts of the host filesystem that running as an app wouldn’t allow - but running Alloy as a “bare” container does.

Beyond that I wanted to explore the latest Grafana features (sparklines in tables!) with some pretty Docker dashboards, which I built with Claude and GCX, the agent-first CLI for Grafana. This allowed me to get a good overview of which container was driving system load and drill into individual containers - and helped me correlate the skipping Music Assistant tracks with CPU spikes. This helped motivate the upgrade to the faster mini PC, which reduced the skipping but didn’t completely eliminate it…

My pretty Docker dashboard... NB aggregate network bandwidth is incorrect due to so many containers running in the host network namespace.

“Layer 2”: Home Assistant

Thankfully this step was much more straightforward, as Home Assistant already exposes decent Prometheus metrics. Enabling these in the config and setting up Alloy to collect the metrics was pretty trivial. My main use for these was collecting long-term, high resolution metrics from a set of cheap Aqara ZigBee temperature sensors I had attached to every radiator in my house with a 3D printed bracket. At first these were constantly dropping off the network, which led me down the path of collecting a ton of Zigbee2mqtt telemetry, but that’s another story…

Once I had these metrics for a few months, I was able to dial in my central heating - tweaking each radiator's valves so they all come up to temperature at roughly the same rate. British central heating systems are weird. When Google decided to screw over Nest customers I moved to Hive, and I went a step further by having my hot water controllable from Home Assistant. This led me to my first Home Assistant PR - adding Prometheus metrics for water_heater domain. Though I wouldn’t recommend Hive - the integration is cloud-based and the auth tokens expire every month or so…

Our central heating during the European heatwave(s)

“Layer 3”: Apps - Unifi & Unpoller

A bunch of people have already put in the work to package existing software up as Home Assistant Apps; one such app is Unpoller, a Prometheus exporter for Unifi’s products. It’s a really great piece of software, and comes with high-quality Grafana dashboards. Installing it and scraping it with Alloy was dead simple - see here

Unifi logs were a bit trickier; Alloy can accept and forward syslog to Loki, but it took me ages to figure out how to configure Unifi to send them. There is an “SIEM Integration” under integrations, that gets you logs for camera detections etc - but to get the firewall logs you need to look under “Network > Settings > CyberSecure > Traffic Logging”. The Alloy config is here. But with these logs I can see which of the devices from my IOT subnet are contacting the internet, and start to lock them down - a topic for a future post.

The excellent bundled Unifi dashboards

“Layer 3”: Apps - Zigbee2MQTT

As I mentioned above, when I started using Zigbee devices I had a particularly unreliable mesh; I tried a bunch of fixes (for another post...) but along the way I added native Prometheus instrumentation to Zigbee2MQTT. There have been attempts at this in the past, and a bunch of projects adding “non-native” (ie instrumenting at the MQTT layer), but I wanted access to internal stats such as queue length, retries, failures etc. This has given me so really useful (and pretty) dashboards:

This is still a work-in-progress; the PRs can be found on GitHub (https://github.com/Koenkk/zigbee-herdsman/pull/1751 & https://github.com/Koenkk/zigbee2mqtt/pull/31645), I’ve built a docker image (docker.io/tomwilkie/zigbee2mqtt-prometheus-amd64:2.12.0-dev) and theres HA app definition on the z2m PR.

Prototype z2m dashboards

Wrapping Up & Next Steps

Getting all my telemetry into one place, and having the history going back months, has helped me dig into the weird and wonderful intermittent 4am failures in my Home Assistant install. This has helped me improve the reliability of the whole setup and in turn improve the WAF. Its also been fun to learn more about how it all works under the covers!

I still need to finish off the z2m PRs. After that I’d love to start collecting traces and profiles of Home Assistant, in particular automations, and see if I can drive down e.g. the latency from motion detection to lights turning on. There also a bunch more telemetry to collect from the various apps I'm using, and other bits of infrastructure I've got. I'd love to get the logs from all my ESPHome devices into Loki..

There have been a ton of "Claude Code is Awesome" posts, and I didn't want to add another one - I wanted to give a concrete example of how it made something that would have taken days (or weeks) take virtually no time at all: building a new integration for Home Assistant.

I have some Norman motorised plantation shutters in my house; I wanted to be able to control them from Home Assistant and integrate them into my evening and morning routines, so they open at sunrise and close at sunset. No integration for these shutters existed for Home Assistant - and while I used to be a full time coder, I'm not proficient in Python and I don't know the internals of Home Assistant. This post gives an overview of the steps I followed to put together what I think is a pretty decent integration…

Step 1: "Draw some circles"

First off I needed to figure out the API and build a client library for them - Home Assistant mandates that all integrations use client libraries instead of directly communicating with the devices. To build this library I needed to know what the API for the shutters looked like. I achieved this by installing the Norman iOS app on my Mac (TIL this was possible!) and used HTTP Toolkit to intercept and log all the traffic. This worked by using the "Anything" option in HTTP Toolkit and configuring a system-wide HTTP proxy on localhost:8000, then running the app and clicking through all the functionality it supports to capture the traffic.

HTTP Toolkit logging the calls from the Norman iOS app.

The HTTP Toolkit has a pro version that allows you to download the captured traffic (I think), but I just manually copied all the paths into a file and prompted Claude Code to build a Python client library against those paths. I explained I wanted zeroconf discovery and it helped me find that the hub advertised a _http._tcp.local. service with the prefix NORMANHUB_.

Over the course of a few days between various other jobs I prompted Claude to "productionize the repo", adding unit tests, linting, CI/CD, pypi packaging, Claude review bot etc. There were a few bugs, in particular around fully opening / fully closing the shutters and the position they reported, but a few rounds of debugging with Claude helped me figure it all out.

The resulting client library is here: https://github.com/tomwilkie/pynormanshutters/. I actually started the project with ChatGPT over 2 years ago, but picked it back up with Claude Code last month when I started on step 2 below. In the meantime I use the pyscript integration to call the client library directly from Home Assistant without building a proper integration…

Step 2: "Draw the rest of the owl"

But calling the client library directly wasn't good enough - I wanted a proper integration so I could do things like e.g. report on the battery status of each shutter. I thought I'd give Claude Code a chance to "one shot it" (I think that's the right term) and gave it a pretty simple prompt:

>Build me a Home Assistant integration for Norman PerfectTilt (https://normanusa.com/) shutters using the pynormanshutters client library. I want a device per shutter, and I want those devices to report open / closed state, angle, and battery charge.

I let Claude go off in plan mode and research how to do it… and the initial version, whilst pretty basic, worked!

The Norman Shutters HA integration

I asked Claude to "productionize" it again, adding CI, tests, linting etc. I asked it to add the Norman logos to the devices, add a device for the Hub and report other entities where available. There were some issues around tilt angle and fully opening / closing the shutters. There were also a couple of trickier bugs that I fixed by prompting Claude with the symptom, having it add logging, and letting it figure out the cause:

Shutters going unavailable. Periodically all my shutters would show as unavailable in HA. The Norman hub returns HTTP 200 with an empty window list when your session expires — no error, just silence. The integration would re-login once, give up, and mark everything unavailable. I described the symptom to Claude, it added debug logging, spotted the cause, and rewrote the update loop with exponential backoff retries.

Autodiscovery silently broken. The integration uses Zeroconf (mDNS) to find the hub automatically on the local network. It worked on my Mac but not on my actual Home Assistant OS install. Turned out there were two separate issues: HA had removed an API field I was using, and the name pattern in manifest.json used uppercase while HA lowercases names on Linux before matching. Two tiny fixes, each basically impossible to spot without knowing HA internals.

To test all of this without breaking my live setup, I also had Claude set up a local Home Assistant instance in Docker.

The resulting integration is here: https://github.com/tomwilkie/ha-norman-shutters and can be installed via HACS.

Summary

This was surprisingly easy, and worked because there was an iOS app for these devices, that I could run on my Mac, and that didn't rely on encrypted traffic. My next project is to repeat this for my alarm system - but I kinda hope that it does more certificate checking…

Hi there! I've had my X9 Omni for ~10 months and loved it so far.

Went to clean it yesterday and noticed the fresh water tank was full, and the dirty water empty. Can't remember last time I filled it, could have been a few months. Looks like the base station isn't dispensing water to either the robot or to wash the mop. Anyone else had this?

I've tried:

• Cleaning everything, replacing the water and asking it to clean the mop manually perhaps 5-10 times.
• Taking the back off and looking for obvious blockages etc
• Disconnecting the fresh water pump and supplying my own 12v - it successfully pushed water through the system into the tray at the bottom.
• Removing the vacuum from the app, turning everything on and off etc - I couldn't find a hard reset method.

Any other suggestions? I have reached out to Ecovacs supports but not heard back yet.

Some photos:

https://preview.redd.it/tiqequz14dzg1.jpg?width=768&format=pjpg&auto=webp&s=e3341374a382ca158582682d2462bdf215636dd8

https://preview.redd.it/tauqh1q44dzg1.jpg?width=768&format=pjpg&auto=webp&s=d88c8752ad2d211bb345654c4c05446f34527fa3