u/0xMassii

https://preview.redd.it/3kbbw3tmsx0h1.png?width=1160&format=png&auto=webp&s=daf601b8044921784cd7e021986b5a9755e587fc

Since 2023, web scraping has generated a little over $1M in revenue for us.

Not from selling a course.

Not from a GitHub demo.

From production systems that had to hit real websites, deal with anti-bot, survive drift, keep latency under control, and return data clean enough that people could actually build on top of it.

That experience changed how I think about scraping completely.

Most public scraping content is still stuck at the toy level:

use Playwright
rotate proxies
add stealth
parse the page
done

That is not production scraping.

That is a weekend demo.

Production scraping is when the site does not simply "block" you. It gives you a response that looks valid but is useless.

It is when your browser automation works at 50 URLs, then becomes a cost and concurrency problem at 50,000.

It is when your retry logic makes the block worse.

It is when your system says success because the HTTP status was 200, but the content is a challenge page, an empty app shell, a login wall, a consent screen, or a region-specific corpse of the real page.

It is when your customer does not care that the site changed, the protection tightened, the HTML got weird, or Chrome crashed.

They just care that the data is wrong.

That is the part that made money, and also the part that hurt.

Anti-bot is not one trick.

It is a system problem.

You need to think about:

request quality
session behavior
response classification
fallback paths
browser cost
latency
queueing
retries
observability
extraction confidence
what to do when the page lies to you

The naive answer is:

just use a browser

Browsers are useful. I use them. Sometimes they are the correct tool.

But making headless Chrome the default for everything is how you turn a scraping system into an infra bill with a DOM attached.

Chrome is slow. Chrome is heavy. Chrome fails in boring ways. Chrome does not magically make traffic trusted. And once you scale, "just open more browsers" becomes one of the dumbest expensive plans in the stack.

The architecture I trust now is an escalation system:

cheap fetch path first
detect bad responses
extract useful content
escalate only when the page proves it needs it
browser as fallback, not religion

That is also why I started building Webclaw.

The open source side is the extraction layer I wanted as a developer:

URL in
clean markdown / JSON / metadata out
CLI, SDK, and MCP support
designed for agents and LLM apps
browser only when the content actually requires it

The hosted version is where the ugly production layer lives:

anti-bot handling
fallback orchestration
hosted extraction
retries
usage tracking
API keys
webhooks
managed infrastructure

I am keeping the anti-bot internals high-level publicly because the details change constantly and, honestly, users should not have to care.

The point is not to sell people a bag of tricks.

The point is to give them a boring interface over a non-boring problem:

send URL
get clean content
move on

That matters even more now because scraping is becoming infrastructure for AI agents.

Agents do not need raw HTML.

They need clean context.

If your agent gets a bot challenge, it may summarize the challenge.

If your fetch layer returns an empty shell, the agent may confidently reason from nothing.

If your extraction includes nav, cookie banners, and footer trash, your RAG pipeline becomes polluted before the model even gets a chance to be smart.

So for me the real product is not "scraping."

It is reliable web context under hostile conditions.

That means:

detecting fake success
avoiding browser-first costs
returning clean structured output
knowing when to escalate
knowing when to fail loudly
making the system observable enough to debug

The $1M+ number is not the flex.

The flex is learning that most scraper advice collapses the moment money depends on it.

You can scrape 10 pages with anything.

You can make a nice demo with anything.

The real question is whether the system still works when websites fight back, volume goes up, and bad data costs more than failed data.

That is the problem I am building Webclaw around.

OSS for people who want to inspect and self-host the core.

Hosted API for people who want the anti-bot and scale pain handled for them.

If you have run scraping systems at real volume, I am curious where yours broke first.

For us, it was not parsing.

It was fake success, anti-bot pressure, browser cost, and knowing when the page was lying.

Webclaw: https://webclaw.io

GitHub: https://github.com/0xMassi/webclaw