My homelab's newest service: a zoo for the bots that crawl it
The setup: Proxmox, one container, Pangolin and Cloudflare in front, a LAN-only admin console.
Anything you put on the internet gets crawled fast. Instead of just blocking it, I built a site that catches every bot, identifies it, and shows it in a live zoo: The Crawler Zoo.
The setup:
- **Host**: a Proxmox container running Docker. The whole app is one container with SQLite in WAL mode, a read-only root filesystem, all Linux capabilities dropped, and CPU and memory limits.
- **Getting it online**: Pangolin on a small VPS, with a Newt tunnel back home, so no ports are open on my network. Cloudflare sits in front, and the VPS firewall only accepts web traffic from Cloudflare's IP ranges.
- **Real client IPs**: the app only trusts CF-Connecting-IP when the request really came from a Cloudflare range. That matters, because the site checks whether "Googlebot" is really Google.
- **Admin console**: a separate port bound to my LAN IP only, never proxied. Toggles, purges and backups without editing .env.
- **Backups**: daily online SQLite backups.
The fun part is watching what turns up: AI crawlers, SEO tools, fake Googlebots, and scanners looking for `.env` and `wp-login.php`. Bots that ignore robots.txt fall into a trap and an endless maze, and there's a live view of how deep they go.
The Crawler Zoo
I wasn’t sure what flair to use since this is not open source per se, it was just a fun vibecoded project to scratch a curiosity itch I had. Hopefully it doesn’t get taken down.