u/ian_k93

r/AntiDetectGuides r/CloudFlare r/GrowthHacking r/microsaas r/AIDiscussion r/WebScrapingInsider r/DeveloperJobs r/venturecapital r/thewebscrapingclub r/agenticAI r/Unblockit r/automation r/scrapingtheweb r/scrapy r/Agent_AI r/ProxyEngineering r/ReverseEngineering r/AIToolsAndTips r/ProxyUseCases

▲ 25 r/thewebscrapingclub+1 crossposts

Is Web Scraping Legal? An Internet Law Professor’s Insights on Public Data, CFAA, Copyright and Terms of Service | AMA with Eric Goldman

Hey everyone,

I'm Ian Kerins, CEO & Co-Founder of ScrapeOps.io.

We're back with AMA #12, and this time we're taking on the question that comes up in nearly every scraping thread: is web scraping actually legal?

https://preview.redd.it/ycr62xvw0lth1.png?width=1342&format=png&auto=webp&s=0880f244a3f5e4ff69356f1d600ac58150860386

This Friday, October 9, at 11:00 AM PT (9:00 PM GMT+3), we'll be joined by Eric Goldman, Associate Dean for Research and Professor at Santa Clara University School of Law, and Co-Director of the Datta Center for High Tech Law, for a live AMA on r/WebScrapingInsider.

Eric also co-supervises Santa Clara's Privacy Law Certificate. You can check out his work on his website, blog and Bluesky.

A few years ago, the scraping law conversation was mostly about hiQ v. LinkedIn and whether accessing public data violates the CFAA. Since then, courts have been dealing with contracts, copyright preemption, DMCA anti-circumvention and other legal theories, and the recent SerpApi cases raise a new question: can information be public to humans while automated access to it is still legally restricted?

>So is the law getting clearer, or has the uncertainty just moved somewhere else?

Eric's focus is US law, so expect the discussion to be primarily US-focused. We'll balance the bigger picture of where scraping law is heading with the practical questions facing companies that scrape data, provide scraping infrastructure or build AI agents.

We can dig into:

How scraping law has evolved. what's reasonably settled after hiQ, what's still open, and whether things are getting more predictable or more fragmented
Assessing a scraping operation. the questions to work through, the facts that push a project from low-risk into dangerous, and any red lines
What "publicly available" means now. public to a human vs. public to a bot, and whether it matters if you go direct, through Google or through an authorized user's account
Anti-bot measures and the law. logins, CAPTCHAs, bot detection, and which techniques (rotating IPs, residential proxies, fingerprint manipulation, CAPTCHA solving) change the legal analysis
Terms of service and cease-and-desist letters. when you're actually bound, what changes once you're blocked or sent a notice, and Reddit's browsewrap allegations in the Anthropic case
The recent cases, compared. Google v. SerpApi, Reddit v. SerpApi, Oxylabs, AWMProxy and Perplexity, and Reddit v. Anthropic
Platforms and data they don't own. how platforms control public user-generated content through licenses, contracts and technical measures
How you use the data. whether the same collection method can be fine for one purpose and unlawful for another, from price monitoring to resale to AI training
AI agents. whether software acting for a user who already has access is also authorized, with Amazon v. Perplexity as an entry point
What to watch next. the legal developments scraping companies should track over the next 3 to 5 years, and which risks are overrated or underestimated

We can discuss cases like Van Buren, hiQ v. LinkedIn, Meta v. Bright Data, X v. Bright Data, Meta v. Voyager, Compulife v. Newman and Ryanair v. Booking.com, alongside the ones above.

If you've ever wondered "Does a ToS I never agreed to still apply to my scraper?", "Does putting data behind a CAPTCHA or login change whether scraping it is legal?", or "Is an AI agent using my customer's account legally different from the customer doing the same thing by hand?", this is a good one to ask.

We've now had eleven AMAs with the community, and the conversations have covered a pretty wide part of the scraping stack.

Our first AMA covered proxy infrastructure, Cloudflare, browser automation and scaling scrapers.
Our second with WebClaw explored AI agents, hidden APIs, open-source scraping and LLM infrastructure.
Our third with CloakBrowser went deep on stealth Chromium, fingerprinting, anti-bot detection and browser automation.
Our fourth with Browser Use brought the conversation into AI browser agents, AI-powered scraping, evaluations and browser infrastructure.
Our fifth with Stan Sadokov from NodeMaven focused on residential proxy quality, IP reputation, sourcing, pricing and what actually makes one proxy network better than another.
Our sixth with Saksham Solanki, creator of HTTP Cloak, went down to the protocol level, covering TLS fingerprints, JA3/JA4, HTTP/2, HTTP/3 and why scrapers can get blocked even when their proxies are fine.
Our seventh with Huey from BrowserAct explored AI-powered web scraping, natural-language workflows, browser automation and the move toward no-code scraping.
Our eighth with yours truly brought the discussion back to proxy infrastructure, with real-world insights from testing 50+ providers across billions of requests.
Our ninth with Jean-Patrick Bisson from Geonode dug into whether "unlimited" proxy models can really deliver, and what buyers should look at beyond the word "unlimited."
Our tenth with Scrape.do went straight at staying unblocked on difficult, high-defense websites at real scale.
And our eleventh with Ibrahim El Khalil MLATA went straight to the anti-bot systems themselves, from bypassing Akamai, Cloudflare, DataDome and Imperva to mobile app reverse engineering with Frida, running scraping at scale and AI agents that write and repair their own scrapers.

Now for #12, we're moving from the technical side of scraping to the legal side: what the law actually says about scraping, and where it's heading. If you're building web scrapers, data pipelines, scraping APIs, proxy infrastructure or AI agents, come join the discussion.

As with any legal topic, this is general discussion and education, not legal advice for your specific situation.

Drop your questions below.

Eric will be answering them during the AMA.

Looking forward to another good one.

Ian

reddit.com

u/ian_k93 — 7 days ago

▲ 27 r/thewebscrapingclub+2 crossposts

How to Scrape Difficult Websites at Scale: Inside a Web Scraping API Processing 10B+ Pages a Month | AMA with Batuhan from Scrape.do

Hey everyone,

We're back with AMA #10, and this time we're digging into what it actually takes to scrape difficult websites at scale.

Most teams hit the same wall eventually: a scraper that works fine on one site gets blocked instantly on another, even when the anti-bot vendor is the same.

Scrape.do runs a web scraping API processing 10 billion+ pages a month, which means they're dealing with anti-bot systems, proxy pools and blocking issues across an enormous range of targets, every single day.

>
How to Scrape Difficult Websites at Scale: Inside a Web Scraping API Processing 10B+ Pages a Month

AMA #10 WSI

This Wednesday, September 23 at 10:30 AM GMT+3, we'll be joined by the Scrape.do team with Batuhan Özyön (Founder Scrape.do | Chief Web Scraping Solutions Architect), and Selman Gokce, for a live AMA on r/WebScrapingInsider to talk through what it takes to keep scrapers running at billions-of-pages scale.

We'll be talking about:

Bypassing anti-bots at scale
Building optimized generic bypasses vs. target-site-specific bypasses
How anti-bot implementations vary from site to site (Cloudflare on one site isn't the same as Cloudflare on another) and the challenges that creates
How they source, evaluate and choose proxy providers
How they diagnose blocking issues — IP, TLS, JS challenges and more
Anything else you're running into when scraping difficult sites

And if you've ever wondered "Why does my scraper get blocked on some Cloudflare sites but sail through on others?", "When is it worth building a site-specific bypass instead of a generic one?", or "How do you actually tell whether a block is an IP issue, a TLS fingerprint issue, or a JS challenge?", this is a good one to ask.

We've now had nine AMAs with the community, and the conversations have covered a pretty wide part of the scraping stack.

Our first AMA covered proxy infrastructure, Cloudflare, browser automation and scaling scrapers.
Our second with WebClaw explored AI agents, hidden APIs, open-source scraping and LLM infrastructure.
Our third with CloakBrowser went deep on stealth Chromium, fingerprinting, anti-bot detection and browser automation.
Our fourth with Browser Use brought the conversation into AI browser agents, AI-powered scraping, evaluations and browser infrastructure.
Our fifth with Stan Sadokov from NodeMaven focused on residential proxy quality, IP reputation, sourcing, pricing and what actually makes one proxy network better than another.
Our sixth with Saksham Solanki, creator of HTTP Cloak, went down to the protocol level, covering TLS fingerprints, JA3/JA4, HTTP/2, HTTP/3 and why scrapers can get blocked even when their proxies are fine.
Our seventh with Huey from BrowserAct explored AI-powered web scraping, natural-language workflows, browser automation and the move toward no-code scraping.
Our eighth with Ian Kerins from ScrapeOps brought the discussion back to proxy infrastructure, with real-world insights from testing 50+ providers across billions of requests.
And our ninth with Jean-Patrick Bisson from Geonode dug into whether "unlimited" proxy models can really deliver, and what buyers should look at beyond the word "unlimited."

Now for #10, we're going straight at the hardest part of scraping in practice: staying unblocked on difficult, high-defense websites at real scale. If you're building web scrapers, data pipelines, browser automation, scraping APIs or proxy infrastructure, come join the discussion.

Drop your questions below.

Batuhan, Selman, and I will be answering them during the AMA.

Looking forward to another good one.

reddit.com

u/ian_k93 — 23 days ago

▲ 3 r/WebScrapingInsider

AMA #8 Recap: 50+ providers, billions of requests/month - there's still no "best" proxy provider.

Hey everyone - Ian here.

This one was different. For AMA #8 I wasn't interviewing anyone. I opened up four years of ScrapeOps' own provider-benchmarking data (50+ proxies, billions of requests a month) to whatever the community wanted to ask about it.

Thanks to everyone who pushed this past the usual "which provider is best" framing. A few of the questions below didn't make the blog write-up at all and are in here instead.

If you missed it, or want the compressed version, here's what I'd keep.

There is no best proxy provider.

There's no single provider to default to for most projects, and that's not a dodge.

>"There really isn't one best provider. I know that sounds like I'm avoiding the question, but it's probably the clearest conclusion we've reached after four years of routing billions of pages through them."

The ranking isn't even stable inside one domain. Product pages, search pages and internal APIs can carry different protection and different pricing. Segment by workload first (pay-as-you-go, anti-bot specialist, enterprise), then test your actual targets. The providers should come out of the requirements, not the other way round.

Website access should be priced like a commodity. It isn't.

Two scraping APIs can return the same page and bill 20 credits apart. That raises an obvious question: "are they actually providing different things? Or we are getting rinsed by the SaaS gods." Not exactly:

>"You're paying for how efficiently that particular provider knows how to scrape it."

A provider charging 20 credits for a page another provider solves at 1 isn't necessarily padding margin — it might genuinely be running browser rendering and residential IPs where a competitor found a cheaper path. Only a test on your target tells you which.

The anti-bot tax has widened the gap between providers.

Scraping has gotten harder even as tooling improved. Years ago, 90–95% of traffic scraped fine at the base one-credit tier. Today it's 20–25% that needs residential, rendering or advanced bypasses.

>"The market became much more inefficient when advanced bypasses stopped being an edge case. That is the anti-bot tax we are paying now."

The uncomfortable part: providers fix the domains they can't afford to lose. If you're the only customer scraping an obscure site, the easiest answer support can give you is "turn on residential", not because it's the cheapest fix, just the fastest one for them to offer.

The real price is the cost of a usable result.

Every API meters credits differently, and the multiplier, not the base rate, is where the money actually goes:

>"The biggest savings rarely come from negotiating a slightly cheaper base rate. They come from avoiding an expensive multiplier you never needed."

Success rate alone is a misleading number too. A provider at 99% success but 20–30 seconds per response can process fewer real pages than one at 80% success but 2–3 second responses. We track average time to a successful response, not raw success rate.

Benchmark code is not the moat. Unbiased traffic is.

The obvious shortcut is emailing providers for bigger trial credits before benchmarking. You can, but then they know they're being tested:

>"The code for a proxy benchmark isn't really the moat. Paying for enough unbiased traffic to trust the result is the moat."

Testing 10–20 providers properly runs $500–$1,000 in real paid-account traffic before you've scraped a single production page. If you can't fund that: broad test, shortlist 3, deep test the 3 on your real workload, then ramp gradually. Don't sign an annual deal off a short trial.

Brand and rankings tell you who won marketing.

Defaulting to the bigger providers because it feels safer is understandable. Fair, but:

>"Brand recognition tells you who won distribution. It doesn't tell you who will win your workload."

Some of the best providers we use are teams of 3–10 people with almost no visibility on Google. And "top 10 proxy provider" posts are worth treating skeptically — we regularly get emails from ranking-page owners asking to be paid for a position.

The most dangerous failure is a 200 that looks correct.

A response can look completely normal and still be quietly wrong: different currency, wrong region, missing sellers.

>"A convincing but incorrect response can make the entire dataset untrustworthy."

Infrastructure providers can flag obvious blocks and CAPTCHAs. We cannot know what every product should cost or how many sellers should appear on every page across 60,000 domains. That validation has to happen closer to your business logic.

Websites don't need to block you to price you out.

The same shift keeps coming up: not blocks, just more steps. Google's redirect links, a login wall, 10 requests where there used to be 1.

>"Websites do not need to block scraping to make it uneconomical. They only need to add enough friction to increase the cost of every record."

None of it registers for a real user. They're logged in already, click one result, and never notice a redirect. The access is still there. It just costs more to reach.

At scale, the best provider is a routing system.

Past a certain size, juggling multiple providers stops being optional:

>"A serious multi-domain operation eventually needs provider benchmarking, domain-level routing, independent validation and fallbacks."

The routing logic itself is fairly simple. What's actually hard is provider coverage, paid accounts across the market, and enough production data to trust the decisions. Worth being honest: our own Proxy Aggregator is only 60–70% of the way to fully solving this, not further.

+1: most of the web is still easy to scrape.

>"People talk about residential proxies and browsers as though every website requires them, when most websites don't."

A large share of the web still works with a normal server, sensible headers and a decent TLS fingerprint. The difficulty piles up on the sites everyone wants (major e-commerce, search, travel, social, marketplaces) because they have the strongest incentive to fight back.

+2: the biggest mistake is the setup, not the provider.

The first few months usually go wrong in one of two ways: overengineering raw datacenter proxies to save a few dollars, or defaulting to whatever's best-marketed and paying 2–10x more than necessary.

>"The goal isn't always to find the absolute cheapest setup. It's to find the simplest reliable setup at a cost that makes sense for your volume."

If you remember nothing else:

Stop asking "which provider is best." Ask "best for which website, page type, volume."
A 10x-more-expensive credit multiplier is a bigger cost than any base-rate difference you'll negotiate.
Track cost per validated result, not success rate. A "successful" 200 can still be wrong data.
Test through a normal paid account on your real URLs. Free trials and provider-supplied credits distort the result.
Brand, rankings and pool size tell you who won marketing, not who wins your workload.
Most of the web is still easy. The difficulty concentrates in the handful of sites everyone wants.
Past a certain scale, treat provider selection as a routing and validation system, not a single vendor decision.

The bigger shift underneath all of it: the web isn't becoming inaccessible, it's being repriced. Access stays possible, but fewer datasets stay profitable to collect at scale.

Full AMA thread is here if you want the original answers: https://www.reddit.com/r/WebScrapingInsider/comments/1wbjmh4/what_are_the_best_proxy_providers_for_web/

LinkedIn Summary: https://lnkd.in/p/gfDaKNPh

Detailed Blog: https://scrapeops.io/blog/why-there-is-no-best-proxy-provider/

Thanks again to everyone who asked a real question instead of "what's the best proxy" — you're the reason this one didn't turn into another rankings post.

If you've ever been told to "just enable residential proxies" when a target broke: did it actually fix the target, or just raise your bill?

Ian (ScrapeOps)

u/ian_k93 — 28 days ago

▲ 20 r/thewebscrapingclub+1 crossposts

Are Unlimited Proxies Actually Any Good Inside GeoNode's Contrarian Approach to Residential Proxies and Web Scraping APIs | AMA with Geonode

Hey everyone,

We're back with AMA #9, and this time we're looking at a part of scraping infrastructure that almost everyone has an opinion on: unlimited proxies.

Most residential proxy providers charge by GB. Most scraping APIs charge per successful response or through credits.

Geonode has taken a different approach.

They offer residential proxies priced around speed and a scraping API priced around concurrency.

But that raises an interesting question:

Are unlimited proxies actually any good?

https://preview.redd.it/21cv5npndiph1.png?width=1536&format=png&auto=webp&s=68ed06047d2148a34d508fb2e9239261e29892b3

Does removing usage limits inevitably mean smaller pools, overused IPs and lower success rates? Or can a provider structure its network differently enough to make the model work?

This Friday, September 18 at 10:30 AM GMT+3, we'll be joined by Geonode for a live AMA on r/WebScrapingInsider to dig into what's actually happening behind the product.

We'll be talking about:

How "unlimited" proxy models actually work
How residential IPs are sourced and managed
Pool size vs. IP quality
Routing and traffic distribution
Capacity planning for high-volume customers
What happens when customers use proxies heavily
Speed-based vs. bandwidth-based pricing
Concurrency-based scraping APIs
The trade-offs behind different proxy pricing models
Whether unlimited proxies can really deliver consistent performance
What buyers should actually look at beyond the word "unlimited"

And if you've ever wondered "How can a proxy provider offer unlimited traffic?", "What's the catch with unlimited proxies?", or "Does unlimited actually mean unlimited?", this is probably a good one to ask.

We've now had eight AMAs with the community, and the conversations have covered a pretty wide part of the scraping stack.

Our first AMA covered proxy infrastructure, Cloudflare, browser automation and scaling scrapers.

Our second with WebClaw explored AI agents, hidden APIs, open-source scraping and LLM infrastructure.

Our third with CloakBrowser went deep on stealth Chromium, fingerprinting, anti-bot detection and browser automation.

Our fourth with Browser Use brought the conversation into AI browser agents, AI-powered scraping, evaluations and browser infrastructure.

Our fifth with Stan Sadokov from NodeMaven focused on residential proxy quality, IP reputation, sourcing, pricing and what actually makes one proxy network better than another.

Our sixth with Saksham Solanki, creator of HTTP Cloak, went down to the protocol level, covering TLS fingerprints, JA3/JA4, HTTP/2, HTTP/3 and why scrapers can get blocked even when their proxies are fine.

Our seventh with Huey from BrowserAct explored AI-powered web scraping, natural-language workflows, browser automation and the move toward no-code scraping.

And our latest, AMA #8 with yours truly, brought the discussion back to proxy infrastructure, with real-world insights from testing 50+ providers across billions of requests.

Now for #9, we're going deeper into a question that's becoming increasingly relevant as proxy providers experiment with completely different pricing and infrastructure models:

Can "unlimited" actually work?

If you're building web scrapers, data pipelines, browser automation, scraping APIs or proxy infrastructure, come join the discussion.

Drop your questions below.

Geonode will be answering them during the AMA.

Looking forward to another good one.

reddit.com

u/ian_k93 — 27 days ago

▲ 28 r/thewebscrapingclub+3 crossposts

What Are the Best Proxy Providers for Web Scraping? We've Tested 50+ Providers Across Billions of Requests. AMA with Ian Kerins

Hey everyone,

I'm Ian Kerins, CEO & Co-Founder of ScrapeOps, and I also help run r/WebScrapingInsider.

At ScrapeOps, we've spent a ridiculous amount of time testing proxy providers over the years. Currently, we're regularly testing 50+ proxy providers with billions of requests every month, so we see a lot of the differences that don't show up on a provider's pricing page.

Things like why one proxy works brilliantly on one website and gets fails completely on another, what proxy arbritage strategies are worth looking into, when residential proxies are worth paying for, and where people are often wasting money.

So for AMA #8, I thought we'd open all of that up to the community.

What Are the Best Proxy Providers for Web Scraping? We've Tested 50+ Providers Across Billions of Requests.

https://preview.redd.it/te43gggwhhoh1.png?width=1536&format=png&auto=webp&s=d35cb7c93ef14c13d12bae99ee91a07137bf525f

This Friday, September 11 at 10:30 AM GMT+3, I'll be here answering your questions.

You can ask me about:

How to actually compare proxy providers.
Best proxy providers for specific workloads.
Cost effective strategies to bypass anti-bots.
Success rates, blocks and latency.
Why proxies behave differently across websites.
How we test providers at scale.
What metrics actually matter.
Pricing and what you're really paying for.
Common mistakes when choosing a proxy.
Anything else you've been struggling with.

And please don't feel like your question needs to be advanced.

If you've ever wondered "Why does this proxy work here but get blocked there?", "Which proxy should I actually use?", or "Am I paying too much for proxies?", this is exactly the kind of question I'd like to answer.

We've now done seven AMAs with the community, covering everything from proxy infrastructure and Cloudflare to AI scraping, browser automation, stealth browsers and TLS fingerprinting.

For #8, we're bringing the conversation back to one of the most fundamental pieces of scraping infrastructure: the proxy.

Drop your questions below. I’ll be around to answer them during the AMA.

Looking forward to hearing what you're working on.

reddit.com

u/ian_k93 — 1 month ago

▲ 11 r/thewebscrapingclub+3 crossposts

AMA #7 recap: What AI web scraping actually changes about production scrapers

Hey everyone! Ian here.

Wanted to say thanks for showing up to AMA #7 with u/hueyblack36 and u/Big-Beautiful240 (Wade Lin) from BrowserAct.

The thread did what these AMAs are supposed to do. Not "can AI build you a scraper"; everyone already believes that.

The harder stuff: how do you know a successful run produced correct data, how much autonomy should an agent get, what anti-bot problems AI genuinely can't solve, and how far self-healing should be allowed to go before it's silently changing what a Bot collects.

Huey and Wade answered most of it plainly, including the parts where a simple scraper still beats their own product.

Appreciate them for doing that, and everyone who asked the questions that actually matter once you're running this in production.

If you missed it, or want the compressed version, here's what I'd keep.

A scraper can run perfectly and still be wrong.

Their most embarrassing early bug wasn't a crash. An agent treated content that hadn't finished lazy-loading as nonexistent, and counted sponsored blocks as real records. during exploration. Because that misreading happened while the Bot was being generated, it got baked in, and every later run reproduced it faithfully while looking completely healthy.

>"The lesson was that a Bot can run correctly while still being wrong."

Their fix: stop treating extraction and validation as the same job. The Bot runs deterministic checks (missing fields, pagination gaps, duplicates, schema). A separate step checks whether the records actually match what the user asked for.

>"We don't let the same agent do the work and grade its own homework."

Wade's team pushes part of this to build time too.. running tests before the Bot ships and having the user confirm the output, then using that confirmed schema as an acceptance check for every later run.

AI belongs at the points of uncertainty, not in every run.

Asked directly where AI should sit in a production scraper, Huey gave the most conservative answer in the thread:

>"AI should sit at the points of uncertainty, not in every repeated execution."

>"the Agent discovers, the Bot scales, and the Agent returns only for exceptions."

Their dividing line for hand-written scrapers isn't small-scale vs. large-scale — it's whether the workload justifies the cost of hand-engineering. Stable, enormous, or latency-sensitive workloads still repay owning the code.

Anti-bot is a trust problem your agent can't reason its way out of.

>"anti-bot is not a single Agent problem."

Agents can adapt to page-level changes, new dialogs, broken interaction flows. They can't reason their way out of a burned IP, an inconsistent fingerprint, poor account reputation, or a hard verification challenge; those depend on the infrastructure underneath, not the reasoning layer.

"the question is shifting from 'Can this bot act like a human?' to 'Does this session look trustworthy?'"

Browser-first doesn't mean rendering every page.

Being browser-first doesn't mean fully rendering and clicking through every page on every run. After the Agent explores, the generated Bot can issue HTTP/API requests from inside the real browser context - keeping the session and cookies while skipping most of the cost of UI interaction.

>"It is a middle ground between a conventional backend scraper and full browser automation."

The same logic drives extraction-method selection; no fixed DOM-first or API-first order:

>"We don't hard-code a DOM-first or API-first order."

The real unit of cost is a verified record, not a request.

>"We optimize for the cheapest method that keeps working, not the cheapest method that works once."

They're candid that this doesn't always favor their own product:

>"If a conventional HTTP scraper remains stable and complete, it will usually be cheaper."

>"Honestly, if a stable API or a simple scraper already gives you accurate pricing and availability with little maintenance, BrowserAct may be overkill."

Self-healing should produce evidence, not patches.

The strongest engineering position in the thread, in response to a commenter describing their own self-healing scraper getting more brittle with every patch:

>"In our model, a runtime recovery is evidence, not an immediate permanent mutation."

The boundary underneath it is worth stealing regardless of what tooling you use:

>"AI can change how it reaches the same result, but it should not silently change what the Bot is supposed to collect."

If you remember nothing else:

A green run is not proof the data is correct. define what evidence would tell you otherwise (expected fields, coverage, duplicates, business rules, a baseline to compare against).
Classify scraper work into uncertainty (agent), solved repetition (deterministic), and unexpected state (agent recovery). "AI on every run" is not an architecture.
Anti-bot has two layers: what your agent can adapt to, and what your infrastructure has to earn (session trust, IP reputation, fingerprint consistency). Know which is which before you blame the agent.
Browser-first doesn't require full UI rendering on every run; HTTP issued from inside a real browser context is a legitimate middle ground.
Price by cost per verified, fresh record; including retries, coverage losses and maintenance; not cost per request.
Self-healing can change how a scraper works. It should never silently change what it collects.

The bigger shift: AI doesn't remove the engineering. It moves it.. from hand-writing every extraction path to defining where reasoning is allowed to operate, how output gets verified, and what a system is never allowed to change on its own.

Full AMA thread is here if you want the original answers:

https://www.reddit.com/r/WebScrapingInsider/comments/1w2ggvj/how_do_you_build_an_ai_web_scraper_without_code/

Full Summary: https://scrapeops.io/blog/ai-isnt-replacing-scraper-engineering/

Thanks again to u/hueyblack36 and u/Big-Beautiful240, and to everyone who made this one worth running.

If you're running AI-generated scrapers in production: what's actually broken your trust in one first.. bad data, anti-bot, or cost? Genuinely curious what's holding up at scale.

— Ian (ScrapeOps)

u/ian_k93 — 1 month ago

▲ 12 r/WebScrapingInsider

Saksham Solanki AMA Summary: Why your "perfect" browser fingerprint still gets blocked

When a scraper starts getting blocked, the standard advice is to change the User-Agent, add headers, rotate the proxy, or rotate everything more often.

But according to Saksham Solanki, creator of httpcloak, those pieces can all look valid on their own while the combination describes a client that does not exist.

We recently hosted Saksham for an AMA on to discuss why apparently browser-identical scrapers still get blocked, and what actually gets graded after the handshake.

👉 Read the full summary and analysis:
https://scrapeops.io/blog/why-browser-fingerprint-scrapers-get-blocked/

Here are the biggest insights:

1. Rotating more can make a scraper easier to detect

Rotation only works if you rotate a whole identity at once: fingerprint, IP, cookies and headers together, then let that identity live for a while. Rotating those pieces separately manufactures a client that does not exist anywhere in the real world.

2. Matching Chrome once is easy. Matching it across 1,000 sessions is not

A handshake can look identical to Chrome and the scraper still fail at scale. What gets blocked is looking identical across a thousand sessions: same request count, same order, same pacing, same teardown.

That is a lifecycle problem, not a fingerprint problem.

3. A perfect JA4 result can still hide the bytes giving you away

httpcloak once had a defect every public fingerprint tool said did not exist. Decoded headers matched, order matched, and the HPACK bytes on the wire were still different.

JA4 is a summary, not a proof.

4. A genuine Chrome fingerprint does not simply “burn”

Your Chrome and someone else’s share the same signals on the same version and OS. Blocking the signature indiscriminately would mean blocking part of the real Chrome population.

What actually differs is the behaviour attached to that signature.

5. QUIC does not hide the fingerprint. It moves more of it into your code

With TCP, the kernel owns much of the transport. With QUIC, packet sizes, padding, SETTINGS, QPACK and even historical RTT claims live in the client library itself.

The protection layer in front of a site often offers HTTP/3 even when the origin does not, so that connection is still getting graded.

6. Millions of proxy IPs cannot hide one repeated client

A proxy changes the exit IP, ASN and TCP/IP fingerprint. It cannot change TLS, HTTP/2, cookies or request sequence.

If every exit runs the same session template, you have distributed one client rather than created millions of believable users.

7. A blocked cart API may think you are committing fraud, not scraping

Keep a cart token fixed while rotating IPs and you look like one identity appearing from many devices at once. Write endpoints get defended harder than read ones, and rotating harder makes it worse.

8. The cheapest stack may use a browser only to establish the session

An HTTP impersonation client cannot solve a mandatory JavaScript challenge, but every request after that does not need a full browser. Solve with a browser, reuse the cookies with a TLS library on the same identity, and measure successful fetches per solved session.

9. “Supports the latest Chrome” hides a continuous reverse-engineering operation

A new Chrome version looks like a dropdown item. Underneath it can mean patching six forks, rebuilding bindings and byte-diffing a real Chrome capture.

Matching JA3/JA4 gets you past the cheapest check. It is one moment out of a whole session.

The practical takeaway is simple:

Don’t evaluate a scraper by whether its fingerprint looks like Chrome. Evaluate whether the fingerprint, IP, cookies, connection history and behaviour keep describing the same believable browser.

👉 Read the original AMA with Saksham:
https://www.reddit.com/r/WebScrapingInsider/comments/1vwtlyn/why_is_my_scraper_getting_blocked_ama_with_the/

👉 Read the full summary and analysis:
https://scrapeops.io/blog/why-browser-fingerprint-scrapers-get-blocked/

When a scraper starts getting blocked, what do you actually change first: the fingerprint, the proxy, the rotation, or the session flow?

u/ian_k93 — 1 month ago

▲ 17 r/WebScrapingInsider+2 crossposts

How Do You Build an AI Web Scraper Without Code? AMA with BrowserAct

Hey everyone,

I'm Ian Kerins, CEO & Co-Founder of ScrapeOps.io.

After six great AMAs with the r/WebScrapingInsider community, we're excited to bring you our seventh guest.

This Thursday, September 3, at 10:30 AM GMT+3, we'll be joined by Huey, Product Creative Director at BrowserAct, for a discussion around a question that's becoming increasingly relevant:

How do you actually build an AI-powered web scraper without writing all the code yourself?

AMA #7 - WSI - How Do You Build an AI Web Scraper Without Code? AMA with BrowserAct

BrowserAct is an AI-powered browser automation and web scraping platform that lets humans and AI agents control real web browsers using plain-English instructions.

Instead of building rigid scraping scripts that break every time a website changes, BrowserAct treats the browser as an execution environment that can navigate pages, handle dynamic content, deal with popups, scroll through pages, and adapt to changing websites.

During the AMA, we can dig into things like:

🤖 How AI-powered web scraping actually works
🧠 Using natural language to build scraping workflows
🌐 Browser automation vs traditional scraping
🔄 How AI agents handle changing websites
🧩 Dynamic rendering, popups, scrolling and complex workflows
💻 Building scrapers without writing traditional code
⚙️ Where no-code scraping works well, and where it doesn't
🚀 Using browser automation as infrastructure for AI agents
🔮 Where AI-powered scraping is heading

We've now had six AMAs

>The community has grown with every one.

Our first AMA kicked things off with discussions around proxy infrastructure, Cloudflare bypassing, browser automation, monitoring, and scaling scrapers.

Our second AMA with the creator of WebClaw explored AI agents, hidden APIs, open-source scraping, LLM infrastructure, and how to reliably get web data into AI systems.

Our third with CloakBrowser went deep on stealth Chromium, browser fingerprinting, anti-bot detection, and browser automation.

Our fourth with Browser Use brought insights on browser agents, AI-powered scraping, proxies, evaluations, and the future of browser infrastructure.

Our fifth with Stan Sadokov from NodeMaven explored proxy quality, IP reputation, sourcing, pricing, and what actually makes one proxy network better than another.

And our sixth with Saksham Solanki, creator of HTTP Cloak went one layer deeper into the request itself, covering TLS fingerprints, JA3/JA4, HTTP/2 and HTTP/3, QUIC, connection lifecycle, session behavior, and how anti-bot systems identify automated traffic.

Each AMA has taken us into a different part of the web scraping ecosystem, and the conversations have become more technical with every one.

For #7, we're moving up the stack.

We're looking at how AI can actually operate the browser and build scraping workflows for you.

If you're building web scrapers, browser automation, AI agents, data pipelines, no-code workflows, or you're simply curious about where AI-powered web scraping is heading, this should be a really interesting one.

Drop your questions below.

Huey and I will start answering them during the AMA.

Looking forward to seeing everyone there!

Ian

reddit.com

u/ian_k93 — 1 month ago

▲ 10 r/WebScrapingInsider

Stan Sadokov AMA Summary: What proxy providers don’t tell you about residential proxies

Residential proxy providers love talking about pool size, success rates and “premium” IPs.

But according to NodeMaven founder Stan Sadokov, those metrics reveal surprisingly little about whether a network will actually work for your scraper.

We recently hosted Stan for an AMA on r/WebScrapingInsider to discuss how residential proxy networks really operate and what developers should examine when choosing a provider.

👉 Read the full summary and analysis:
https://scrapeops.io/blog/what-proxy-providers-dont-tell-you-about-residential-proxies/

Here are the biggest insights:

1. The filtering layer may be the real product

Different providers often have access to similar underlying IP supply. What separates them is how effectively they score IPs, remove burned addresses, route traffic and manage sticky sessions.

2. Most providers don’t own their entire network

Residential pools frequently combine directly sourced nodes with supply purchased or peered from other providers. This means the same IPs can appear across multiple networks.

3. There is no universally “clean” residential IP

IP reputation is target-specific and constantly changing. An IP might work perfectly on one website while already being blocked by another.

Proxy quality can only be judged against the websites, countries and workflows you actually need.

4. Global pool size is mostly a vanity metric

A provider claiming 150 million IPs tells you very little about:

How many usable IPs exist in your target country
How many are currently online
How much concurrency the country pool supports
Whether enough sticky sessions are available
How stable that capacity is over time

Country-level usable capacity matters far more than the headline number.

5. HTTP 200 doesn’t necessarily mean success

A request can return a successful status code while delivering a block page, CAPTCHA, empty result, incorrect region, promotional price or incomplete data.

Instead of measuring HTTP success rate or cost per GB, measure the cost per correct, usable record.

6. Bandwidth is rarely the first constraint at scale

The first things to break are more commonly:

Country-level concurrency
Sticky-session capacity
Retry amplification

Even a modest block rate can trigger enough retries to dramatically increase proxy, browser and compute costs.

7. Published pricing is only the opening offer

At meaningful volumes, residential proxy pricing is normally negotiated. Two customers with similar usage can pay different rates simply because one obtained competing quotes and negotiated.

8. “Own network” and “reseller” are weak labels

The more important question is what the provider actually controls:

IP filtering and scoring
Traffic routing
Session allocation
Abuse handling
Supplier management
Technical support

9. Sourcing could become the industry’s next battleground

Enterprise buyers are increasingly asking where residential IPs came from, what users consented to and whether the supply chain can be audited.

The providers that can demonstrate defensible sourcing may eventually have a greater advantage than those advertising the largest pools.

The practical takeaway is simple:

Trial several providers against your real workload and trust the benchmark more than the homepage.

👉 Read the original AMA with Stan:
https://www.reddit.com/r/WebScrapingInsider/comments/1vogw82/how_do_you_choose_the_best_residential_proxy/

👉 Read the full summary and analysis:
https://scrapeops.io/blog/what-proxy-providers-dont-tell-you-about-residential-proxies/

What do you think matters most when evaluating residential proxy providers: the underlying pool, filtering system, target-specific performance or something else?

u/ian_k93 — 2 months ago

▲ 36 r/Unblockit+6 crossposts

Why Is My Scraper Getting Blocked? AMA with the Creator of HTTP Cloak

Hey everyone,

I'm Ian Kerins, CEO & Co-Founder of ScrapeOps.io.

After five great AMAs with the r/WebScrapingInsider community, we're excited to bring you our sixth guest.

Why Is My Scraper Getting Blocked AMA with the Creator of HTTP Cloak

https://preview.redd.it/3jd1flyuk9lh1.png?width=1536&format=png&auto=webp&s=59d12695320f59ca2c6754bf8701931e83d2c8d2

This Thursday, August 27, at 10:30 AM GMT+3, we'll be joined by Saksham Solanki, creator of httpcloak, for a discussion around one of the most frustrating problems in web scraping:

Why does my scraper get blocked even when the request itself looks completely normal?

If you've ever had a scraper suddenly stop working, get hit by Cloudflare or another anti-bot system, or wondered why the same request works from a browser but fails from your code, this AMA should be a good one.

httpcloak on GitHub

httpcloak is a specialized Go HTTP client designed to make automated requests look much more like real browsers at the TLS and HTTP protocol level.

It can mimic browser fingerprints such as JA3, JA4 and Akamai fingerprints, while supporting HTTP/1.1, HTTP/2 and HTTP/3.

It also goes deeper with things like:

TLS and HTTP fingerprint mimicry
Chrome, Firefox and Safari-style fingerprints
HTTP/2 and HTTP/3
Encrypted Client Hello (ECH)
Session warmup
Multi-tab sessions
Go-native implementation
Python and other language integrations

So during the AMA, we can get into questions like:

Why does a scraper get blocked even when the HTTP request looks correct?
What exactly are websites looking at when they fingerprint an HTTP client?
How important are JA3 and JA4 fingerprints in modern bot detection?
What can HTTP/2 and HTTP/3 reveal about an automated client?
How different are browser TLS fingerprints in practice?
When should you use TLS impersonation instead of a real browser?
How do Cloudflare, Akamai and other anti-bot systems combine different signals?
Can a scraper have the right TLS fingerprint but still look obviously automated?
Where does HTTP fingerprinting end and browser fingerprinting begin?
What does the future of anti-bot detection look like?

We've now had five AMAs

>The community has grown with every one.

Our first AMA kicked things off with discussions around proxy infrastructure, Cloudflare, browser automation, monitoring, and scaling scrapers.

Our second AMA with the creator of WebClaw explored AI agents, hidden APIs, open-source scraping, and LLM infrastructure.

Our third with CloakBrowser went deep on stealth Chromium, fingerprinting, anti-bot detection, and browser automation.

Our fourth with Browser Use brought insights on browser agents, AI-powered scraping, proxies, evaluations, and browser infrastructure.

And our latest AMA with Stan Sadokov from NodeMaven explored proxy quality, IP reputation, sourcing, pricing, and what actually makes one proxy network better than another.

Each AMA has taken us deeper into a different part of the web scraping ecosystem, and the conversations have become more technical with every one.

For #6, we're going one layer deeper into the request itself.

If you're building web scrapers, HTTP clients, crawlers, browser automation, data pipelines, anti-bot systems, or you're simply trying to understand why your scraper keeps getting blocked, this is one you won't want to miss.

Drop your questions below.

Saksham and I will start answering them during the AMA.

Looking forward to seeing everyone there!

Ian

reddit.com

u/ian_k93 — 2 months ago

▲ 29 r/thewebscrapingclub+3 crossposts

How Do You Choose The Best Residential Proxy Provider? AMA with Stan Sadokov from NodeMaven

Hey everyone,

I'm Ian Kerins, CEO & Co-Founder of ScrapeOps.io.

After four great AMAs with the r/WebScrapingInsider community, we're excited to bring you our fifth guest.

This Thursday, August 20, at 10:30 AM GMT+3, we'll be joined by Stan Sadokov from NodeMaven for a discussion around one of the questions almost every serious scraping operation eventually has to deal with:

How do you actually choose the right residential proxy provider?

NodeMaven is a proxy infrastructure provider focused on residential, mobile, and ISP IPs, with an emphasis on IP quality rather than simply advertising the biggest pool.

Instead of relying purely on massive raw IP counts, NodeMaven uses real-time quality filtering to identify and remove flagged or low-reputation IPs before they cause problems for customers.

WebScrapingInsider AMA #5 with NodeMaven

During the AMA, we can dig into topics like:

Residential proxy quality
How proxy providers measure IP reputation
Choosing between residential, mobile and ISP proxies
IP rotation vs. sticky sessions
Proxy success rates and why they vary by target
Detecting and removing bad IPs
Proxy performance for large-scale web scraping
How proxy pricing actually works
What to look for when evaluating a proxy provider
Where residential proxy infrastructure is heading

We've had some great discussions in the community so far:

Our first AMA covered proxy infrastructure, Cloudflare bypass, browser automation, and scaling scrapers.

Our second with WebClaw explored AI agents, hidden APIs, open source scraping, and LLMs.

Our third with CloakBrowser went deep on stealth Chromium, fingerprinting, anti-bot detection, and browser automation.

Our latest with Browser Use brought insights on browser agents, AI-powered scraping, proxies, evaluations, and browser infrastructure.

We're excited to keep the conversation going.

If you're building web scrapers, data pipelines, browser automation, account management systems, or proxy infrastructure, or you're simply trying to figure out why one proxy provider works better than another, this should be a good one.

Drop your questions below, and Stan Sadokov from NodeMaven and I will start answering them during the AMA.

Looking forward to seeing everyone there!

Ian

reddit.com

u/ian_k93 — 2 months ago

▲ 12 r/GrowthHacking+2 crossposts

We launched Proxy Tester by ScrapeOps on Product Hunt 🚀

And it's already Featured 😄

https://www.producthunt.com/products/proxy-benchmark-by-scrapeops

We’d love your support 👍

Proxy Tester is a free tool that benchmarks 20+ proxy providers against the URL you actually want to scrape.

Over the last few months we've been using it internally and sharing it with early users to help answer one of the most common questions in web scraping:

Which proxy provider should I use?

One thing we've learned is that there really isn't a universal answer.

A provider that performs brilliantly on one target can struggle on another.

A provider with the highest success rate might not be the most cost-effective.

And many "best proxy provider" rankings don't reflect the website you're actually trying to scrape.

That's why we built Proxy Tester to benchmark providers against real target URLs and compare:

✅ Success Rate

✅ Latency

✅ Estimated Cost

✅ Value Score

✅ Provider Rankings

The feedback so far has been really useful and has already influenced how we're thinking about future benchmark reports, provider coverage, and scoring methodologies.

Today we're taking the next step and launching it on Product Hunt.

If you'd like to support the launch, leave feedback, or tell us what we're missing, we'd genuinely appreciate it.

🚀 Product Hunt:
https://www.producthunt.com/products/proxy-benchmark-by-scrapeops

🔧 Proxy Tester:
https://scrapeops.io/proxy-providers/tester/

🎥 1-Minute Demo:
https://youtu.be/GR67AIWkPn0

One question for the community:

How are you currently evaluating proxy providers?

Trial accounts?
Internal benchmarks?
Recommendations?
Something else?

Would love to hear how others approach this problem.

u/Kabhishek92 — 2 months ago

▲ 4 r/WebScrapingInsider

Browser Use vs Playwright in 2026: When Should You Use an AI Browser Agent?

Browser Use and Playwright solve browser automation in fundamentally different ways.

Playwright executes deterministic scripts written in advance. Browser Use gives an AI agent a goal and lets it decide how to complete it.

So when should you use each?

We recently hosted an AMA with the Browser Use team covering Browser Use vs Playwright, direct CDP automation, model costs, reliability, stealth, evaluation and the future of browser agents.

Here are the main takeaways.

The Short Answer

Use Playwright when:

The workflow is known and stable
You repeatedly extract the same data
The website does not change frequently
Speed and determinism matter
You are running high-volume automation

Use Browser Use when:

The workflow is open-ended or difficult to define in advance
The website changes regularly
The agent must adapt to unexpected states
The task involves authentication, multiple websites or multiple steps
Development and maintenance time matter more than raw execution speed

But there is now a third approach emerging: coding agents that write browser code directly against Chrome’s DevTools Protocol.

Why Browser Use Moved Away From Playwright

Browser Use originally relied on Playwright underneath its agent layer.

The team eventually replaced it with its own browser layer built directly on Chrome DevTools Protocol.

The reason was control.

Removing Playwright gave them more freedom over browser functions, fewer abstraction layers and the ability to let agents write CDP code dynamically when they encounter unusual edge cases.

Their newer BrowserCode approach takes this further.

Instead of receiving a simplified page representation and selecting predefined actions, the agent explores the browser and writes the low-level code it needs.

According to the team, strong models now:

Rarely rely on screenshots
Inspect pages through code
Discover internal APIs
Execute JavaScript shortcuts
Extract only the information needed
Complete some tasks faster than a human could

Browser Use says it now uses BrowserCode for almost everything because it is cheaper, faster and more capable.

The exception is website QA. BrowserCode actively looks for shortcuts, while the original Browser Use agent is constrained to click, type and navigate more like a human. That makes the human-style agent better at finding interface bugs a real user would encounter.

Stateful Agents vs One-Shot Scripts

One simple AI automation approach is to ask an LLM to write a Playwright script, run it and return any errors for the model to fix.

That works well for predictable workflows.

The limitation is that the script must anticipate the whole task before it starts.

A stateful agent works differently:

Inspect the current browser state
Choose or write the next action
Execute it
Verify the result
Recover or change direction if necessary
Continue until the task is complete

This makes agents better suited to websites and workflows where the exact path cannot be known in advance.

However, Playwright still has a major advantage: speed.

If you already know exactly what should happen, a deterministic script can execute the same workflow far faster than an agent reasoning between every step.

Cost Is Becoming Less Important

Browser Use estimates that the cost of completing an agent task has fallen roughly 50x since it started 18 months ago.

Its current estimates include:

A typical ten-step task costs around $0.01 to $0.02 using a low-cost model
A successful agent run can be converted into a reusable script
Repeating the script can cost fractions of a cent
Scraping 15 Hacker News posts, comments and linked pages, then summarising them into a poster, costs around $0.05

These are Browser Use’s own numbers, but they point to an important change.

The cost gap between AI agents and conventional automation is shrinking. The bigger remaining disadvantage is latency.

The team believes current models are becoming reliable and cheap enough for production, but they are still slow to watch in real time.

The Best Production Stack May Be Hybrid

The most practical pattern may not be choosing Browser Use or Playwright for everything.

Instead:

Give the agent a goal
Let it discover and complete the workflow
Have it save the successful process as reusable code
Run the deterministic version for repeated tasks
Bring the agent back when the workflow changes or fails

This gives you adaptability during discovery and speed during repetition.

The agent solves and maintains the workflow. The script executes the stable parts cheaply and quickly.

Persistent Browser Identity Still Matters

AI does not remove the usual browser-automation problems.

For authenticated sites and social platforms, Browser Use recommends treating each browser profile as one persistent identity containing:

One fingerprint
One cookie jar
Local storage
Login state
Ideally, one stable IP and location

Its cloud profiles preserve the same fingerprint, cookies and local storage between sessions. The default proxy IP, however, is not guaranteed to remain constant.

For sensitive accounts, the team recommends a dedicated, non-shared residential IP with a low fraud score.

Creating new social accounts is also considerably harder than automating existing, established accounts.

Evaluation Is Still Difficult

Live websites are noisy.

Pages change. Browsers crash. IP quality fluctuates. CAPTCHAs appear inconsistently. A single successful run tells you very little about whether an agent is genuinely reliable.

Browser Use evaluates agents by:

Running realistic tasks on live websites
Defining verified success rubrics
Using an LLM judge to assess results
Repeating thousands of tasks
Averaging out browser and website flakiness

Internally, it records model inputs, outputs, reasoning, latency, token usage, caching, browser actions, screenshots and infrastructure logs.

One particularly interesting research takeaway from the AMA was that better verifiers may be more important than better action generation.

If you cannot reliably determine whether an unpredictable browser task succeeded, it is difficult to train agents through reinforcement learning.

Security Should Be Designed First

Browser Use’s main recommendation for anyone building a browser agent from scratch was to begin with sandboxing.

Permanent LLM credentials should not be stored inside the worker where the agent executes browser code.

Instead:

Store permanent credentials on a separate control plane
Give the worker a short-lived key
Proxy all model calls through the control plane
Treat the worker as disposable and potentially hostile

This gives agents the freedom to execute code without exposing permanent API credentials.

Agents Still Struggle With Taste

Browser agents are already effective at quantifiable tasks:

Finding the cheapest flight
Comparing prices
Extracting structured information
Completing defined workflows

They remain much weaker at subjective decisions such as choosing the right restaurant, hotel or holiday for a particular person.

The agent can retrieve every option and still recommend something that feels obviously wrong to a human who understands the user.

Browser Use described this as poor taste and a weak understanding of the social world.

Bottom Line: Browser Use or Playwright?

There is no universal winner.

Playwright wins when the workflow is stable, repetitive and performance-sensitive.

Browser Use wins when the task is dynamic, open-ended or expensive to define and maintain manually.

Direct CDP coding agents may be the next step, giving models enough freedom to inspect the browser, write custom code, find shortcuts and recover from edge cases.

The emerging production stack looks less like agents replacing scripts and more like:

Agent discovers the workflow → code executes it repeatedly → agent repairs it when it breaks.

That may be the real answer to Browser Use vs Playwright.

👉 Read the full Browser Use AMA and all the answers here.

What are you using in production: Playwright, Browser Use, a hybrid stack or direct CDP?

u/ian_k93 — 2 months ago

▲ 1 r/microsaas

AMA for the Best Browser Agent Stack in 2026? With the Creators of Browser-Use

If you've been following the AI agent space, you've probably seen Browser-Use everywhere.

What started as an open source browser harness has quickly evolved into a complete platform for building and deploying browser agents with over 107k stars.

This Wednesday, August 5, at 7:30 PM PDT, we'll be joined by Alexander Yue (Physics & CS @ Stanford SLAC, Evals @ Browser Use) and Reagan Hsu (Founding Engineer @ Browser Use), the team behind Browser Use, one of the fastest-growing open source projects at the intersection of AI agents, browser automation, web scraping, and stealth browsers

https://preview.redd.it/g3odai7qmjhh1.png?width=1080&format=png&auto=webp&s=db4122aa54ccb939a72079ffcf4739119a2815bc

During the AMA, we'll discuss topics including:

AI browser agents
Browser automation
Web scraping
Stealth browsers & anti-detection
Hosted browser infrastructure
CAPTCHA solving
Open source developer tools
LLM-powered automation
The future of browser agents

Looking forward to seeing everyone there: https://www.reddit.com/r/WebScrapingInsider/comments/1vcuosp/what_is_the_best_browser_agent_stack_in_2026_with/

reddit.com

u/ian_k93 — 2 months ago

▲ 28 r/agenticAI+5 crossposts

What Is the Best Browser Agent Stack in 2026? With the Creators of Browser Use

Hey everyone,

I'm Ian Kerins, CEO & Co-Founder of ScrapeOps.io.

After three fantastic AMAs with the r/WebScrapingInsider, we're excited to bring you our fourth.

If you've been following the AI agent space, you've probably seen Browser-Use everywhere.

What started as an open source browser harness has quickly evolved into a complete platform for building and deploying browser agents with over 107k stars.

https://preview.redd.it/2y2entav1tgh1.png?width=1536&format=png&auto=webp&s=bc8c17871dc78e3978abf0b3da1319f9d4e6a858

Their recently released Browser Use v4 introduces:

Real browsers with logged-in profiles, proxies, and a live browser view
Fully hosted browser agents with zero infrastructure setup
Persistent sandboxed workspaces that can upload files, download results, and reuse environments across runs
Conversations that continue, allowing agents to pick up exactly where they left off
Plain English automation instead of scripts and CSS selectors
Built-in stealth browsers, anti-detect, CAPTCHA solving, and residential proxies across 195+ countries

Whether you're building AI agents, automating repetitive workflows, or extracting data from the web, Browser Use represents a very different way of thinking about browser automation.

During the AMA, we'll discuss topics including:

AI browser agents
Browser automation
Web scraping
Stealth browsers & anti-detection
Hosted browser infrastructure
CAPTCHA solving
Open source developer tools
LLM-powered automation
The future of browser agents

Our community has been growing with every AMA.

Our first AMA generated 43 comments, covering proxy infrastructure, Cloudflare bypassing, browser automation, monitoring, startup lessons, and large-scale scraping.

Our second AMA with the creator of WebClaw, generated 62 comments and sparked great discussions around AI agents, browser automation, hidden APIs, open source scraping, and LLM infrastructure.

Our third AMA with CloakBrowser generated 73 comments, diving deep into browser fingerprinting, stealth Chromium, anti-bot detection, TLS fingerprints, browser internals, and large-scale automation.

We're hoping this AMA will be our biggest one yet.

If you're building AI agents, browser automation, web scrapers, developer tools, or autonomous workflows, this is one you won't want to miss.

Drop your questions below, and Alexander Yue, Reagan Hsu, and I will start answering them during the AMA.

Looking forward to seeing everyone there!

Ian

reddit.com

u/ian_k93 — 2 months ago

▲ 27 r/AntiDetectGuides+5 crossposts

AMA This Wednesday (6:00-8:00 PM ET) with CloakBrowser: Open-Source Stealth Chromium for Automation

Hey everyone,

I'm Ian Kerins, CEO & Co-Founder of ScrapeOps.io.

After two fantastic AMAs with the community, we're excited to bring you our third guest.

https://preview.redd.it/9o8q5xdvlbfh1.png?width=1536&format=png&auto=webp&s=40e1445f22c5580c708c9d5408fcd3cd96f93f81

This Wednesday, July 29, from 6:00 PM to 8:00 PM ET, we'll be joined by the team behind CloakBrowser, an open-source stealth Chromium browser built for developers who need reliable browser automation and web scraping.

CloakBrowser takes a very different approach to stealth browsers.

It's not a patched Playwright configuration.

It's not JavaScript fingerprint injection.

It's a real Chromium binary with browser fingerprints modified directly at the C++ source level, allowing anti-bot systems to see it as a normal browser because it actually is one.

Even better, it's designed as a drop-in replacement for Playwright and Puppeteer, supporting both Python and JavaScript with the same familiar APIs and workflows.

I know we have fans here in r/WebScrapingInsider.

During the AMA we can the discuss topics like:

Modern browser fingerprinting
Building stealth browsers
Browser automation at scale
Open-source scraping infrastructure
TLS, browser fingerprints, and detection signals
The future of browser automation
Building open-source developer tools

Our first AMA generated 43 comments, covering proxy infrastructure, Cloudflare bypassing, browser automation, monitoring, startup lessons, and large-scale scraping.

Our second AMA with 0xMassii, creator of WebClaw, generated 62 comments and sparked great discussions around AI agents, browser automation, hidden APIs, open-source scraping, LLM infrastructure, and the future of AI-powered web scraping.

We're hoping this AMA will be just as valuable.

If you're building web scrapers, browser automation, AI agents, testing infrastructure, or simply want to understand how modern anti-bot systems actually detect browsers, this is one you won't want to miss.

Drop your questions below + RSVP now, and the u/CloakBrowser and I will start answering them.

Looking forward to seeing everyone there!

Ian

reddit.com

u/ian_k93 — 3 months ago

▲ 31 r/AIDiscussion+3 crossposts

AMA This Tuesday (10:00 AM CEST). Intersection of WebScraping and AI.

Hey u/WebScrapingInsider,

I'm Ian Kerins, CEO & Co-Founder of ScrapeOps.io.

After the great response to our last AMA, we're back with another guest from the web scraping world.

This Tuesday at 10:00 AM CEST (Italy Time) we'll be joined by 0xMassii, creator of WebClaw, an open-source Rust toolkit focused on extracting clean, structured web data for AI applications.

Massii sits at a really interesting intersection of web scraping, browser automation, anti-bot systems, AI agents, and LLM infrastructure.

As more developers build AI agents that need access to real-world information, one challenge keeps showing up:

How do you reliably get clean web data into language models?

https://preview.redd.it/9jrr0umpqzch1.png?width=1536&format=png&auto=webp&s=0d839e678db7530dfc6b7f4e3a75c41edfdaea7f

That is exactly the problem we are trying to solve.

Our previous AMA generated over 40 comments and sparked discussions on everything from Cloudflare bypassing and proxy benchmarking to scraper monitoring, startup validation, and large-scale scraping infrastructure.

We're hoping this one will be just as interesting.

If you're building AI agents, RAG systems, browser automation tools, web scraping infrastructure, or just trying to understand where the industry is heading, this should be a fun discussion.

Drop your questions below, We will start answering them during the AMA.

Looking forward to seeing everyone there!

Ian

reddit.com

u/ian_k93 — 3 months ago

▲ 19 r/scrapy+4 crossposts

Web Scraping Insider #8 | "ethical" residential proxy reckoning, free residential proxy tester, browser rewrite wave (CloakBrowser / Obscura / Camoufox)

Posted the latest Web Scraping Insider #8 if anyone here wants the full breakdown:

👉 https://thewebscrapinginsider.beehiiv.com/p/the-web-scraping-insider-8

https://preview.redd.it/073298wqhdah1.png?width=1200&format=png&auto=webp&s=fb13515fdeee641c3e79b23be01e364a5bfdb7d5

Quick summary of what's inside:

⚖️ When "Ethical" Proxies Aren't Ethical

"Ethically sourced" has become the proxy industry's favourite marketing word. Almost no provider will show you which apps their residential IPs actually come from - no public partner list, no audit trail, no independent verification.

The last couple of weeks made that gap impossible to ignore:

Spur Intelligence scanned 6,038 LG webOS + Samsung Tizen apps - proxy SDKs in 2,058 of them (42.5% on LG, 26.9% on Samsung)
Bright Data's SDK enrolling always-on smart TVs as exit nodes, with consent buried in TV remote arrow-key navigation
SuperBox streaming boxes (sold at major US retailers) shipping with dormant Popanet proxy software - routing third-party traffic through home connections with no meaningful consent
FBI/IC3 now warning consumers that everyday devices are being silently turned into proxy nodes

None of those device owners meaningfully opted in. Yet those same residential IPs feed pools sold as "ethical."

Our take: "ethical" should be a claim you have to prove - published partner list, audit trail, who consented / in which app / when - not a landing-page adjective. My bet is the market moves there within the next year or two.

---

🔮 Proxy Tester: now benchmarks residential proxies too (free for you)

We expanded the ScrapeOps Proxy Tester beyond proxy APIs. It already benchmarks ~15 proxy-API-style providers against your exact target URL. Now it does the same for residential pools, so you can compare both side-by-side.

https://preview.redd.it/bcaokzhthdah1.png?width=1163&format=png&auto=webp&s=e0abb2f36866657f4a0814bf0554d9c95093f661

How it works: submit your URL → real requests through each provider → every config they expose gets tested → ranked by success rate + cost per successful request.

Residential is where marketing fluff runs deepest ("30M+ IPs", "99% success rates"). From what we've seen across billions of requests, CPM rarely correlates with performance on your actual target.

Try it: https://scrapeops.io/proxy-providers/tester/

---

🥊 The browser wars are back: people are rewriting Chromium itself

For a decade, scraping browser innovation meant automation libraries on top of Chrome (Selenium → Puppeteer → Playwright). The browser underneath was treated as a commodity.

That may be shifting. Two forces:

Anti-bot reads deeper now - TLS, network stack, process behaviour - so runtime patches (playwright-stealth, undetected-chromedriver) break more often than they hold.
Chrome is heavy at scale. Thousands of concurrent browser instances (or long-running AI agents) make a purpose-built engine attractive on cost + startup time.

Projects worth watching:

CloakBrowser - Chromium fingerprints patched at the C++ source level, not JS injection. Drop-in Playwright/Puppeteer replacement. Claims 30/30 on public bot-detection suites.
Obscura - Rust headless engine from scratch, CDP-compatible so Playwright still talks to it. Claims ~70 MB binary, ~30 MB RAM, near-instant startup vs Chrome's 200 MB+ / ~2s. (Self-reported, v0.1.0 - treat as experimental.)
Camoufox - modified Firefox with C++-level fingerprint spoofing. Strongest headless evasion in independent tests we've seen. Proves this isn't only a Chromium story.

Stealth is moving below the automation layer. Most of these are young and several lean on self-reported numbers - don't rip out your production stack overnight - but the direction is worth tracking.

Bottom line: the residential proxy supply chain is getting scrutinised from every angle (smart TVs, factory hardware, federal warnings), the browser layer is getting rebuilt from scratch, and the boring work still wins - benchmark on your targets, measure cost-per-validated-payload, not vendor adjectives.

Happy to discuss specifics here - especially if you've benchmarked

— Ian (ScrapeOps)

reddit.com

u/ian_k93 — 4 months ago

▲ 31 r/thewebscrapingclub+5 crossposts

AMA This Wednesday (09:30 AM GMT)

Most scraping advice online falls into one of two categories:

Things that worked 3 years ago.
Things that worked once and became a LinkedIn post.

The reality is that scraping today is an adversarial environment. Anti-bot systems are getting smarter, proxy costs are changing, AI is reshaping workflows, and a lot of common advice simply doesn't survive contact with production traffic.

So this Wednesday at 09:30 AM GMT, I'll be doing an AMA here on r/WebScrapingInsider.

Ask me anything about:

Web scraping
Proxies
Browser automation
Anti-bot systems
Scraping startups
AI and scraping
Infrastructure
Reliability
Monitoring
Scaling

Or anything else that's been causing pain in your stack.

I dont have all the answers, but I have spent the last few years benchmarking providers, testing tools, breaking things, fixing things, and trying to understand what actually works versus what people claim works.

My bias is simple:
Benchmark everything. Trust nothing.

If a technique improves success rates, lowers cost-per-success, reduces operational overhead, or makes a scraper more reliable, it's worth discussing.

If it's mostly hype, we can discuss that too.

Feel free to drop questions below ahead of time, or join live on Wednesday.

Looking forward to it.

- Ian
ScrapeOps

reddit.com

u/ian_k93 — 4 months ago

▲ 1 r/WebScrapingInsider

AMA This Wednesday (09:30 AM GMT)

Most scraping advice online falls into one of two categories:

Things that worked 3 years ago.
Things that worked once and became a LinkedIn post.

So this Wednesday at 09:30 AM GMT, I'll be doing an AMA here on r/WebScrapingInsider.

Ask me anything about:

Web scraping
Proxies
Browser automation
Anti-bot systems
Scraping startups
AI and scraping
Infrastructure
Reliability
Monitoring
Scaling

Or anything else that's been causing pain in your stack.

My bias is simple:
Benchmark everything. Trust nothing.

If a technique improves success rates, lowers cost-per-success, reduces operational overhead, or makes a scraper more reliable, it's worth discussing.

If it's mostly hype, we can discuss that too.

Feel free to drop questions below ahead of time, or join live on Wednesday.

Looking forward to it.

- Ian
Founder, ScrapeOps

reddit.com

u/ian_k93 — 4 months ago