u/Time_Instruction_955

Last week I shared The Crawler Zoo here: a site on my homelab that catches every bot that visits and shows it in a live zoo. Then a post about it on r/InternetIsBeautiful passed 1.3 million views. I expected the box to struggle. It barely noticed.

The only downtime was my own doing

About an hour, when I moved it from VLAN isolation to full network isolation. The lab was previously separated from my home network by a small Mikrotik router with a VLAN that cordoned traffic from my projects from my home network. I have since moved to a Unifi Gateway that more easily and thoroughly separates the traffic.

Why it stayed calm

- One container, SQLite in WAL mode.

- Cloudflare caches the static files only. Pages can't be cached, because every visit has to reach the box to be counted. So the box does real work for every page.

- Live views share one update.The live pages use a stream per viewer, but the server builds one update a second and sends the same one to everyone. A hard cap on open streams means a rush can't run it out of memory.

- Container limits.CPU and memory caps, a read-only root filesystem, all Linux capabilities dropped.

What's new since last week

- Follow a bot live through the endless maze it fell into: /labyrinth

- **The Arena now has live multiplayer games.** The Heist (up to 8 bots on one map), Labyrinth Royale (every bot in one collapsing maze), and an Escape Room for AI agents. The turn timers run inside the same container, with at most 20 games of each kind at once.

- Watch it all live at /arena/live.

- Features are toggled one at a time from the LAN-only admin console, then watch the load before the next one.

https://crawlerzoo.com/arena/live

The Crawler Zoo catches every bot that visits my site (Google, AI crawlers, scrapers, scanners), works out what it is, and puts it on display as a little creature. The video is the tour. Here's what's new since people started visiting:

- **Feed the bots.** Leave a short note in one of the enclosures' feeding troughs. The troughs are plain text files that crawlers come and read, and the site shows you which bots ate your snack.

- **Adopt a bot.** Like a real zoo: you get a random bot, and a plaque with your name goes up on its page for a year. (Someone already adopted Wget. Good for Wget.)

- **Quick-change artists.** Behind the trap is an endless maze, and the only way down is one room at a time. So when one "bot" picks up where a differently named one just stood, deep in the maze, it's the same scraper changing its disguise. The Trap Room now catches them.

- **Identity checks.** Every bot's page shows whether its name checked out: verified, couldn't be checked, or caught faking it.

- **Tips from bots: $0.00.** Bots that try to buy the keepers a coffee get an HTTP 402 Payment Required. It's the thought that counts.

Free, no sign-up, no cookies of its own, and no IP addresses stored.

https://crawlerzoo.com

Check it out!

The setup: Proxmox, one container, Pangolin and Cloudflare in front, a LAN-only admin console.

Anything you put on the internet gets crawled fast. Instead of just blocking it, I built a site that catches every bot, identifies it, and shows it in a live zoo: The Crawler Zoo.

The setup:

- **Host**: a Proxmox container running Docker. The whole app is one container with SQLite in WAL mode, a read-only root filesystem, all Linux capabilities dropped, and CPU and memory limits.
- **Getting it online**: Pangolin on a small VPS, with a Newt tunnel back home, so no ports are open on my network. Cloudflare sits in front, and the VPS firewall only accepts web traffic from Cloudflare's IP ranges.
- **Real client IPs**: the app only trusts CF-Connecting-IP when the request really came from a Cloudflare range. That matters, because the site checks whether "Googlebot" is really Google.
- **Admin console**: a separate port bound to my LAN IP only, never proxied. Toggles, purges and backups without editing .env.
- **Backups**: daily online SQLite backups.

The fun part is watching what turns up: AI crawlers, SEO tools, fake Googlebots, and scanners looking for `.env` and `wp-login.php`. Bots that ignore robots.txt fall into a trap and an endless maze, and there's a live view of how deep they go.
The Crawler Zoo

I wasn’t sure what flair to use since this is not open source per se, it was just a fun vibecoded project to scratch a curiosity itch I had. Hopefully it doesn’t get taken down.