Your firewall might be blocking AI crawlers while your browser shows everything is fine.
Ran into something on a recent project that's worth sharing because it's easy to miss.
Client was in business brokerage. Site looked completely normal. Pages loaded, Google was indexing content, HTTPS set up, firewall active. Standard checks came back fine.
But when I fetched the site the way an AI crawler would, I kept hitting:
"JavaScript is required."
The firewall was challenging non-browser requests. Which meant AI crawlers couldn't reach the actual page content at all. They were hitting security wall, and getting nothing.
The fix was straightforward once we found it: adjust the firewall rules to allow legitimate AI crawler user agents, clear the firewall cache, then test again without a browser.
That last part matters. Opening the site in Chrome and seeing it work proves nothing here. You have to test it the way a crawler sees it, not the way a human does.
This is one of those issues that hide perfectly because your browser tells you everything is fine while crawlers are getting blocked entirely.
It also point to something broader. We used to ask "can Google crawl this?" Now the question is "can the different AI system actually access and understand it?" Those are not the same audit.
Expecting to see a lot more of this as AI visibility become a standard part of technical SEO work.
Anyone else running into firewall and security layer issue specifically affecting AI crawler access?