u/blimy20

So I am one of the seemingly young(ish) real ale drinkers around here.

Trying to nail down the gems. Share or up vote any that are worth adding.

- Nags Head

- The Fox (Caversham)

- The moderation.. Only a couple of pulls but I am a Sussex Best lover.

- Castle Tap.

What am I missing? Ideally ones that have 4+ on tap.

We spent a while researching this properly instead of going off vibes. Every engine wants something slightly different, and the blockers matter more than any of the on-page tricks people usually talk about.

ChatGPT wants OAI-SearchBot allowed and the answer stated plainly in your first couple hundred words. Blocked by that crawler being disallowed (different from GPTBot, which is training only) or by content that only exists after JavaScript renders.

Perplexity wants outside proof you exist, mostly Reddit and other community threads, plus something recently published. Blocked by PerplexityBot being disallowed, or a page with zero third-party corroboration anywhere.

Claude wants you findable in Brave Search specifically. Anthropic uses Brave as a backend and one independent check found ~79% overlap between what Claude cites and Brave's own results. Blocked by a blanket robots.txt disallow, since that quietly takes out Bravebot too.

Google AI Overviews and Gemini want normal, competent SEO plus an actual point of view, nothing AI-specific. Blocked by noindex/nosnippet, or Googlebot getting blocked at the CDN rather than in robots.txt.

Same pattern everywhere: the crawler split (training bot vs search bot) trips more people up than anything about the content itself.

So a potential show stopper here to AI visibility if you use Cloudfare. Our robots.txt allowed every AI crawler explicitly, by name but still had crawler issues.

Cloudflare's managed AI-bot setting was returning a 403 at the edge, before anything reached the file. Ran like that for weeks in June. Nothing errored and nothing alerted and there's no gap in analytics to spot it because a crawler that gets turned away doesn't show up as anything at all. It just looks like a quiet month.

The check that caught it: fetch your live robots.txt over the public internet, not the copy in your repo. If the first line is a comment block about conditions of access instead of your own directives, something upstream is rewriting it. Then request a normal page with a crawler user agent from outside your network and confirm you get a 200 and not a challenge.

Worth doing even if you're sure it's fine. I was sure it was fine... but bam!

So a potential show stopper here to AI visibility if you use Cloudfare. Our robots.txt allowed every AI crawler explicitly, by name but still had crawler issues.

Cloudflare's managed AI-bot setting was returning a 403 at the edge, before anything reached the file. Ran like that for weeks in June. Nothing errored and nothing alerted and there's no gap in analytics to spot it because a crawler that gets turned away doesn't show up as anything at all. It just looks like a quiet month.

The check that caught it: fetch your live robots.txt over the public internet, not the copy in your repo. If the first line is a comment block about conditions of access instead of your own directives, something upstream is rewriting it. Then request a normal page with a crawler user agent from outside your network and confirm you get a 200 and not a challenge.

Worth doing even if you're sure it's fine. I was sure it was fine... but bam!

Quick disclosure, I build an AI visibility scanner, so this is me marking my own homework in public. Our scorer used to award points for having llms.txt. It doesn't now. but why...

The embarrassing bit is that our own playbook had been saying the file does nothing for months while the scorer carried on rewarding it. Nobody joined the dots.

What finally did it wasn't the studies, though those are damning enough on their own. Someone's run the numbers across a few hundred thousand domains now and there's no relationship there at all. Plenty of sites have had the file up for ages, so you can't really fall back on "too early to tell" either. And Google's own docs say it outright, "Google Search itself doesn't use them."

It was realising retrieval never touches the file. Engines pull from a search index, ranked on relevance and trust. You can't publish something on your own server and have it turn up in someone else's index. That's not how any of it works, same reason meta keywords stopped mattering.

Keep the file if you've already got one. Ten minutes, harmless, a couple of small tools do parse it. Just don't let an agency put a line on your retainer for maintaining it.