r/WebForAI

Mayhaps you've read it alreadt, but Patreon partnered with Cloudflare's AI Crawl Control. In their own testing, once blocking went live, weekly attempts from training crawlers dropped from thousands to zero. In my eyes, this kiind of confirms what most people already assumed, robots.txt was never enforced, it was just a request most crawlers ignored when it suited them. Notable part is they're not blocking everything. Indexing bots that send traffic back still get through, it's specifically training crawlers getting cut off. Cloudflare launched Pay Per Crawl last year and just changed default policy this month so mixed use crawlers (index + train) get blocked by default on ad-supported pages. Add in the NetNut/Popa takedown and the Bright Data SDK stuff and it feels like robots.txt as a soft suggestion is on its way out, at least for anyone with the budget to enforce something stronger. Now, for AI agents, this mostly targets the RAG/training use case, scraping content to store, retrain, or reuse later. Agents that fetch a page live to answer a question and link back to Patreon should be fine, since that's the indexing behavior they explicitly still allow. Agents that pull content into a knowledge base or fine-tuning pipeline get treated like training crawlers and blocked. Net effect: builders relying on naive scraping will need better infra (proper agent identification, residential proxies, proper fingerprinting.)

It's easy to assume that if a page loads and the text extracts cleanly, the data is good enough to feed into a model. That's what majority thinks. What I have in mind is duplicate pages with slightly different info, structured data that doesn't match what's rendered on screen, inconsistent formatting across sites that breaks parsing logic downstream. Can relate? What's been the hardest data quality issue to catch once an AI agent started sourcing from the web instead of a clean dataset, and how did you end up dealing with it?

I've been at this company for five years now, most of that time as one of the core engineers, and things changed practically overnight. Management decided a few weeks ago that we're going all in on agentic AI for internal workflows, and I actually support that in principle. What I can't get behind is the process they've suddenly rolled out. The mandate came down from non-technical management with basically no warning. The new expectation: prompt an AI coding tool to generate the entire backend, agent logic, tool definitions, skills, all of it, then engineers just adjust it until the thing behaves. There's no code review requirement anymore. Nobody has to actually understand what the generated code is doing before it goes out. Security is supposedly the infra team's job, but that only covers network, auth, deployment. Nothing about the application layer, prompt injection through tool outputs, agents with permissions that are way too broad, data crossing user boundaries that shouldn't be crossed. None of that gets caught because reading the code isn't part of the process anymore. After five years of doing this the normal way, a few things are really sitting wrong with me. Nobody owns the code anymore. When something breaks in production down the line, I'm the one who's going to be reverse-engineering logic that no human, including whoever "wrote" it, ever actually understood. And personally, I've spent these years building real engineering judgment at this company. Watching that get replaced overnight with "just trust the output" feels like a gut punch. I'm completely fine with AI-assisted development where I still understand and own what ships. I'm not fine with this. So, questions for anyone who's lived through a similar shift: has anyone managed to push back successfully on a sudden requirement from non-technical managers, and what actually worked? Does the fact that this happened suddenly, after years of a completely different standard, change how seriously I should take it as a red flag? And for people in AI-heavy dev shops, where do you personally draw the line between real AI-assisted engineering and just vibecoding?