We blocked a page in robots.txt then asked 8 AI assistants to read it
I put a page on a test server, blocked it in robots.txt for every AI user agent I could find a name for, then went and asked eight assistants to open it and tell me the code printed on it. Also kept a second page unblocked so I could tell refusing apart from just never fetching.
ChatGPT, Claude and Meta all refused. Claude even throws a proper error, ROBOTS_DISALLOWED, "Site disallows automated access." Gemini, Grok and Manus all just read it anyway. Copilot and Perplexity refused to even check, so I got nothing useful from those two.
I was curious on Gemini, so I searched and found Google's docs saying user-triggered fetchers generally ignore robots.txt, because a person asked for it. So it's deliberate. .
Grok didn't even ask for robots.txt. Not once, in three rounds.
One domain and three rounds, so take it for what it is, and it's only about fetches a user triggers, nothing to do with training crawlers.