u/shasedoge

what worked a year ago feels like it's falling off fast, so curious what people are actually running now.

for the sites that really fight back, cloudflare, datadome, the nastier akamai stuff, are you just going straight to a browser now or still trying http first with curl_cffi and only spinning up a browser when you have to? and honestly i can never decide how much of it is proxy quality vs just having your client set up right. feels like i blame the wrong one half the time.

not fishing for anyone's secret setup or anything, just trying to figure out if there's some rough consensus on what a sane setup looks like in 2026 or if everyones still just duct taping their own thing per site.

Trying to get a read on where people actually draw the build-vs-buy line in 2026, because it feels like it's moved.

for context, the tradeoff as I see it: rolling your own gives you control and lower per-request cost, but you eat all the maintenance when targets change, anti-bot shifts, proxies get flagged. paying for a scraping API or managed service hands off the headache but the cost scales with volume and you're stuck with whatever they support.

what I'm curious about from people running this at real scale:

where's your actual cutover? like, is it volume, is it how hostile the target is, is it just not having the headcount to maintain scrapers?

for the stuff you still build yourself, what makes it worth keeping in-house?

and for anyone who went the other way and moved off managed APIs back to building, what pushed you back?

Mostly wondering if the "just build it" default has shifted now that the managed options got better, or if at scale it still always comes back to owning your own stack.

what are you all actually listening to for scraping / data eng?

went looking for this in the sub and the last real thread on it was over a year ago, so figured it's worth asking again, the space moves fast enough that half those recs are probably stale.

mostly after stuff that gets into the weeds on:

• scraping and anti-bot cat-and-mouse
• data pipelines and the infra behind them
• proxies / web data collection

doesn't have to be scraping-specific either, some of the best episodes I've heard on this were one-off guests on broader data eng shows. open to those too.

what's actually worth the subscribe in 2026?

“Rotating IPs” and “rotating identity” often get treated like they’re the same thing.

They’re not, and that’s usually why “clean” IPs still end up getting blocked. IMHO.

Modern anti-bot systems look at much more than your exit IP. They correlate cookie continuity, header order, TLS fingerprints, request timing, session behavior, and even the way requests fail. So if you swap to a fresh IP but keep using the same python-requests TLS handshake and the exact same headers you’ve sent thousands of times before, you’re not really hiding anything.

You changed one signal while everything else stayed the same.

A real identity is the whole package: IP and geo, cookies, headers, TLS fingerprint, request pacing, session length, retry patterns, and overall behavior. Keep those things consistent within a session, but let them vary naturally between sessions.

Each identity should look like one real user from start to finish.

One mistake I see a lot is rotating the IP halfway through a session. That breaks your own continuity and suddenly your “user” jumps halfway across the world. That’s a pretty obvious signal on its own.

My rule of thumb is:

> Static residential or ISP IPs for login-based or long-lived sessions where you want a stable identity.

> Rotating pools for broad, non-login scraping where each identity is disposable.

> Sticky sessions for multi-step workflows, long downloads, or anything that depends on continuity.

One thing that’s worth pointing out: the proxy only gives you the network layer, meaning your IP, location, and session persistence. It can’t make your identity believable on its own. Your application still has to manage cookies, tokens, headers, and behavior consistently for each identity. When those two layers line up, a lot of the “my IP is clean but I’m still blocked” problems disappear.

I’m curious how far people actually take this in practice.

Do most of you template headers and TLS per identity, or do you find that’s overkill unless you’re dealing with particularly aggressive targets?

I’m still learning this stuff, so if I’m missing something obvious, I’d love to hear how more experienced people handle it.

Every "why do my free proxies keep dying" thread gets the same reply: just pay for proxies. which is lazy and kind of misses the point. free lists are fine for messing around, learning, a one-off pull. the trouble starts when you put them in production, and it's not the price that gets you.

the stuff that actually costs you time:

you can't tell a block from a dead proxy. request fails, and you have no idea if the target blocked you or the exit just vanished. so every failure turns into a debugging session. that's the real tax, way more than the proxies themselves.

success rate that won't sit still. free exits blink in and out, so you end up writing all this retry logic just to cover for supply you can't count on. you're basically engineering around your proxies instead of the site you're trying to scrape.

no idea what the IP's been doing. it might already be burned on your exact target before you send a single request, and you'd never see it coming. you're inheriting someone else's mess.

you don't know who's running it. someone owns those exits. no clue if they're logging what goes through them. anything with a login or a sensitive target, that alone should be a no.

the thing that flipped it for me: stop counting cost per GB or per IP. count cost per successful request, and throw in the hours you spend babysitting. a free list pulling 30% with constant hand-holding is more expensive than a boring paid pool that just works. you're just paying in time instead of money.

and no, free isn't always wrong. prototyping, learning, quick pulls, totally fine. but the second something runs non-stop and you actually care if it works tomorrow, known and accountable beats free, doesn't matter who you buy from.

where's your cutover, though? when do you actually give up on free exits and pay? feels like most people hang on way too long.

Every couple weeks someone hits a wall scaling Playwright or Puppeteer and decides the answer is more browser instances. Usually it isn't.

The browser isn't your scaling unit. If you're rendering every request you're paying full render cost on a pile of pages that never needed JS in the first place.

What actually scales is hybrid. Browser only for the stuff that genuinely needs it, login, token init, the heavy SPA pages. Pull the cookies and tokens out of that session, then hand the boring repeatable traffic to plain HTTP workers. Queue it, cache it, retry logic on top.

So one browser session bootstraps auth, then a pool of lightweight HTTP workers reuses that state behind your exits. Browser count stays flat while throughput climbs. The expensive part drops to maybe 5% of the job instead of being the whole job.

Past 100k requests a day the bottleneck is almost never "not enough browsers." It's queue design, retry budget, and whether you're actually reusing sessions or re-authing constantly.

The other half of this is routing. Not every request needs the same exit either. Login and session traffic wants a stable identity, bulk list/detail pages can ride cheaper paths, media and file pulls want bandwidth. Sending all of it through one flat setup is where a lot of cost and blocks come from.

Curious where people draw the browser line. I keep moving more stuff to HTTP-first than I used to, but there's a class of sites where the token refresh is annoying enough that it's not worth the fight.

What's still forcing a full browser for you?

Hey everyone,

Looking for podcast recommendations from people actually working in the field.

Specifically interested in:

• Web scraping and anti-bot
• Data engineering and pipelines
• Proxy infrastructure and web data collection

What are you listening to?