u/kilee_geo

So I was auditing my own site's robots.txt today and noticed something kind of wild.

A lot of sites — especially ones that haven't touched their robots.txt in a few years — are blocking GPTBot, ClaudeBot, PerplexityBot by default. Not on purpose. Just because those bots didn't exist when the rules were written and the catch-all `Disallow` is doing its thing.

Like I scanned a Shopify memorial gifts store the other day. Score was 35/100. Turned out GPTBot was blocked and they had zero llms.txt. Fixed both, score went to 86. That was maybe 30 mins of work total.

For anyone who doesn't know — llms.txt is basically a plain text file at yoursite.com/llms.txt that tells AI models who you are, what you do, how to cite you. Think robots.txt but for LLMs. Without it the model is just guessing from your HTML and honestly the guesses are not great.

Quick things to check:

- yoursite.com/robots.txt — ctrl+f for GPTBot. not there? probably blocked.

- yoursite.com/llms.txt — if it 404s you don't have one

Not saying this is going to 10x your traffic tomorrow but AI referral traffic is growing and it costs basically nothing to fix. Figured I'd share since I haven't seen many people talking about the robots.txt issue specifically.

Anyone else been looking into this stuff?

AI search is changing how people discover brands. ChatGPT has 2.8B monthly users. Perplexity is growing 45% QoQ. And AI-referred visitors convert at 4.4x the rate of organic search.

But most brands — including well-known ones with strong SEO — are completely invisible in AI search.

After researching this for months, I found 3 specific reasons why:

1. robots.txt is blocking AI crawlers (GPTBot, ClaudeBot, PerplexityBot) — most sites do this accidentally

2. No llms.txt file — AI has no structured briefing on the brand, so it guesses (and often gets it wrong)

3. Incomplete schema markup — AI can't confidently understand what the business does

The fix for #1 takes 5 minutes. The others take a bit longer but are totally doable without a developer.

I wrote up the full breakdown with examples and step-by-step fixes. Drop a comment if you want the link — happy to share.

Happy to answer questions — this stuff is still pretty new and there's a lot of confusion around it.

I've been working on this for a while and finally launched it.

The idea came from noticing that AI search (ChatGPT, Perplexity, Claude) is growing fast, but almost no sites are set up for it. Most are accidentally blocking AI crawlers or have no llms.txt file — so AI engines simply don't know they exist.

I built foundinai.co to diagnose this and help fix it. It scans a site and gives an AI visibility score in about 60 seconds.

I've tested it on some well-known brands:

• Glossier: 50/100 (above average, but missing structured data)
• CeraVe: 41/100 (below average, AI crawlers blocked)

Honest questions I'm trying to figure out:

1. Does this problem even matter to you as a site owner?
2. Is the score useful or just noise?
3. What would make you actually pay for something like this?

Not trying to pitch — genuinely want to know if I'm solving a real problem or barking up the wrong tree.

Nike is a $50B brand. They spend millions on SEO. But their AI Visibility Score? 59/100.

Here's what AI search engines can't find on Nike.com: - No llms.txt file (AI bots are literally guessing what Nike sells) - robots.txt is blocking GPTBot and ClaudeBot - No product schema on individual product pages

Industry average for e-commerce is 48. Nike scores 59 — above average, but leaving massive AI traffic on the table.

The fix? Less than 2 hours of work. This is the gap between traditional SEO and AI search optimization.

Anyone else noticing AI referral traffic showing up in GA4? Curious what others are seeing.

Been running AI visibility audits on well-known DTC brands. Glossier was one of the most interesting cases.

The score: 50/100. Barely above the beauty industry average (48).

What's actually broken (technical breakdown):

**robots.txt** — blocking several major AI crawlers including Perplexity and Claude. Their content is literally invisible to these platforms.

**JSON-LD schema** — missing on product pages. ChatGPT and Gemini can't parse what products they sell, what category they're in, or what the brand stands for.

**Alt text** — 85% of product images have none. AI vision models use alt text as a primary signal for product understanding.

**No llms.txt** — the newer protocol (like robots.txt, but for LLMs) that tells AI crawlers how to interpret your site. Zero adoption among big beauty brands so far.

The irony: Glossier's growth was entirely word-of-mouth. AI search IS the new word-of-mouth. And they're structurally invisible to it.

For SEOs here — are you seeing clients ask about this yet? Curious how much AI search visibility is showing up in briefs.

Nike is a $50B brand. They spend millions on SEO. But their AI Visibility Score? 59/100. Here's what AI search engines can't find on Nike.com: - No llms.txt file (AI bots are literally guessing what Nike sells) - robots.txt is blocking GPTBot and ClaudeBot - No product schema on individual product pages Industry average for e-commerce is 48. Nike scores 59 — above average, but leaving massive AI traffic on the table. The fix? Less than 2 hours of work. This is the gap between traditional SEO and AI search optimization. Anyone else thinking about AI search optimization for their business?