llms.txt adoption vs actual crawler behavior, anyone tracked GPTBot/ClaudeBot hits against a llms.txt file directly?
Been pulling raw access logs across a few client sites to see which AI crawlers actually respect llms.txt versus just ignoring it and crawling everything anyway. GPTBot and Google-Extended show up consistently in the logs, but I can't tell from the log data alone whether they're reading the llms.txt directive first or just crawling regardless and the file is functionally decorative right now.
Also noticing a pattern where sites with clean schema (FAQ, HowTo) and short, direct-answer paragraphs early in the content seem to get cited more often in AI Overviews and Perplexity answers, but I don't have a controlled way to prove causation versus just correlation with sites that already had strong technical SEO.
Has anyone run an actual before/after test on this, adding llms.txt and structured Q&A formatting to a page, then tracking crawler behavior and citation frequency over a few weeks? Trying to figure out if this is a real lever or if it's cargo-culting a spec most crawlers don't fully honor yet.