u/Individual-Court-217

I will not promote- Building one-click protection for your websites against AI. Would love to just learn!

I've just been intrigued by how the web is changing for the past couple months with Ai agents accessing the web more than humans and learning where the breakdown happens with this shift. The web to be very honest is just not made for agentic actions right now. Whether it's buying products, signing up accounts, signing up for mailing lists, etc, the web is just not built for it yet.

I crawled 1m domains to see how website owners are looking at this shift in terms of machine readable policies, protection, etc. Almost 90% of the websites have nothing and are just hoping for the best.

I wanted to give website owners back control of what ai agents can do on their websites with setups within a couple clicks irrespective of what platform you use, vercel, wordpress, aws, etc. You would be able to allow whether an agent can transact or only browse, can search your webpage but not train on it.

I'm mostly trying to talk to real site owners across categories - whether you've had agents doing weird stuff on your site (accidental purchases, scraped pricing, bandwidth spikes, fake signups, training data, whatever) or whether you've thought about it and just don't know where to start!

Happy to answer anything!

reddit.com
u/Individual-Court-217 — 13 days ago
▲ 2 r/saasbuild+1 crossposts

Building one-click protection for your websites against AI. Would love to just learn!

I've just been intrigued by how the web is changing for the past couple months with Ai agents accessing the web more than humans and learning where the breakdown happens with this shift. The web to be very honest is just not made for agentic actions right now. Whether it's buying products, signing up accounts, signing up for mailing lists, etc, the web is just not built for it yet.

I crawled 1m domains to see how website owners are looking at this shift in terms of machine readable policies, protection, etc. Almost 90% of the websites have nothing and are just hoping for the best.

I wanted to give website owners back control of what ai agents can do on their websites with setups within a couple clicks irrespective of what platform you use, vercel, wordpress, aws, etc. You would be able to allow whether an agent can transact or only browse, can search your webpage but not train on it.

I'm mostly trying to talk to real site owners across categories - whether you've had agents doing weird stuff on your site (accidental purchases, scraped pricing, bandwidth spikes, fake signups, training data, whatever) or whether you've thought about it and just don't know where to start!

Drop a comment or shoot me a dm, happy to answer anything!

reddit.com
u/Individual-Court-217 — 13 days ago
▲ 4 r/nocode+1 crossposts

Built a scanner to check every known AI policy standard across the

Tranco top 1M domains. Here's what came back:

- 90.1% of domains have no AI policy at all (no robots.txt directives for AI bots, no llms.txt, no ai.txt, nothing)

- 7,575 sites prohibit AI scraping in their Terms of Service but don't enforce it technically. The agents see no restriction in robots.txt, the legal terms say stop. ToS gap.

- 6,317 sites have conflicting signals across files. robots.txt allows GPTBot, llms.txt blocks it, etc.

- 29 AI bots blocked by nytimes.com alone

Everyone's still living on robots.txt rules from the 1990s.

For folks here who've actually dealt with AI bot traffic - what worked for you? Just block-all in robots.txt? Cloudflare? Custom WAF rules? Did your hosting bill drop after blocking?

Also curious whether you've seen agent-driven traffic that wasn't just crawling - anything that looked like it was actually trying to do things on your site (form submissions, account creation, scraping pricing in real-time)?

u/Individual-Court-217 — 26 days ago