r/ProxyEngineering

juggling multiple ad accounts and social media profiles for clients and my current setup is letting me down. accounts keep getting flagged and i need something more reliable.

been reading up on antidetect browsers paired with residential proxies as the go-to solution for this. makes sense in theory but not sure which combo actually holds up in practice. currently testing proxyshard for residential proxies and shard browser for the antidetect side. will report back about how its going

what are you guys using for multi-account work? specifically curious about which antidetect browser you pair with your proxies and whether residential or ISP proxies work better for staying stable long term

curios if anybody in this community is successfully scraping ticketmaster or axs for their event data. both are notoriously hard targets and consistently challenge heavier scrapes. im newish to the proxy game so maybe i am not really aware of whats out there.

The conversations here keep circling back to proxy selection. Which provider, which type, sticky vs rotating. But I have been watching fingerprinting detection evolve since 2015 and the proxy is rarely what causes a working setup to degrade over weeks.

What actually kills sessions is coherence failure.

Platforms do not score individual signals. They score whether your signals agree with each other. A residential IP says "regular consumer in Berlin." Your browser's TLS fingerprint says "modified Chromium." Your canvas hash says something else entirely. No single signal is the problem. The pattern is.

I analyzed over 100 threads on fingerprinting and account bans. The failure narrative is almost always the same: day one works fine, then something quietly accumulates, then everything falls apart at once. The account was not killed by one bad check. The trust score eroded because the signals stopped telling a coherent story.

A few things that changed how I think about this:

Detection starts at the transport layer. If your TLS fingerprint marks the connection as automated before the page loads, the JavaScript checks do not matter. The decision is already made.

Proxy to profile coherence matters more than proxy quality. A residential IP paired with a browser profile that expects a different locale is not a coherent setup. It is two contradictory signals pointing in opposite directions.

Randomization is not the answer. A fingerprint that randomizes every field contradicts itself. What survives over time is a setup where every layer tells the same story.

The question worth asking is not "which proxy provider should I use." It is "what story are all my signals telling together, and do they agree?"

Curious if others have mapped out which signal mismatches tend to surface first in their setups?

Ok so this is gonna be a bit of a ramble but I need y'all to hear this. I work adjacent to the scraping/data space (not naming employer, y'all know how it is) and I always kind of just you know, accepted that "residential proxies" were this slightly grey but mostly fine thing. Like yeah obviously somebody's home IP is being used, somebody agreed to something somewhere, moving on. Then yesterday I see netnut is just gone. Not down, not maintenance page, straight up FBI DOJ IRS-CI seizure banner, Google and Lumen and Shadowserver all stamped on it too like it's a whole coalition operation. Like bruh, I have never in my life seen this on any website. So imagine the shocker lol. And I'm sitting there like wait since when did the IRS care about proxy IPs??? Turns out IRS-CI does financial crime investigations and it could be related to that, but still, seeing that logo on a proxy provider's main website??? Diabolical mate.

So I go searching what's going on and apparently it's tied to this thing called the Popa botnet, basically it's been running for like four years hijacking Android TV boxes (remember that Bright Data SDK thingy?? And Krebs traced a chunk of it back to actual NetNut infrastructure through some ex-employee's domain. Not some randoms, an actual former VP of R&D there. Google's own blog post says they think the network was over 2 million devices at one point and that a lot of those "different" proxy brands you see around are just NetNut wearing a different logo through resellers. I swear I seen a post talking about how there is one or two proxy providers sharing those IPs/reselling whatever. Makes sense now, people?? Am I the only one concerned here with a few other nerds? And to be fair I want to be fair here, Alarum (the parent company) came out and started calling it that a botnet is inaccurate, also said that there's real KYC and consent flows and misuse detection on their end. So it's not like this is some minor thing everyone agrees on, it's actively disputed and I think that matters, I'm not trying to just slam a company because Krebs wrote a scary headline, no. Make what you will out of it, but I things are rising to the surface, more and more often. Again it does make me think about how much of the "residential proxy pool" any of us are touching day to day is actually consented in a way a normal person would recognize as consent, versus consented in the "buried in paragraph 14 of an SDK terms screen" way. Like where's the line between legit residential network and just a nicer branded botnet, and how would any of us even know the difference from the outside.

Not trying to start a witch hunt on the whole industry, genuinely just spent my evening reading legal filings and threat intel blogs instead of sleeping like a normal person, but after Bright Data SDK shennanigans and now Netnut, boy, there's gonna be more stuff coming out. So I figured I'd share since I know a bunch of you here actually work with this infra daily and probably have opinions on these matters.

Sources if anyone wants to go check themselves.

Krebs on Security, the original Popa botnet reporting: https://krebsonsecurity.com/2026/06/popa-botnet-linked-to-publicly-traded-israeli-firm/

Google's threat intel writeup on the disruption: https://cloud.google.com/blog/topics/threat-intelligence/google-continued-disruption-residential-proxy-networks

Alarum's official response: https://alarum.io/alarum-technologies-responds-to-inquiry-into-residential-proxy-networks/

And divinetworks.com itself, which is also showing the seizure page now: https://divinetworks.com/

P.S Some time later I found that they mixed up the domains because .com is down, but not .io which is the main page of netnut.

Hey peeps, perhaps someone is scraping betting sites odds, such as lsports, draftkings, betonline, bovada etc? Looking for the most optimal way of doing so.

https://preview.redd.it/lvon1ku4uzah1.png?width=574&format=png&auto=webp&s=f4a099056b6a54236c4bdb4dc0b827ef70813051

I was inspired by today's news

Been testing different setup lately and I feel like proxy alone is not enough anymore. Residential and mobile proxy can work but using it together with a cloud phone seems more stable for me. Even if the IP is clean, account can still have issues when the device, location, or activity keeps changing. Feels like the full environment matters now, not just the proxy.

For people managing multiple accounts, what do you prioritize more now, better proxy or stable environment?

Basically the title. If you have anything to share regarding automations, please do, looking to alleviate majority of manual day to day work.

ok so this is gonna be long sorry in advance, but I spent my precious weekend comparing n8n scraping workflows against just writing the damn scraper in Python and I have some thoughts to share with yall.

Started because my unemployed friend sent me one of those "I automated my job search with n8n" posts and I was like, not with this again, there's like a million automations already created, why did you even bothered? But he somehow convinced me to try replicating something similar on my side, so basically, I had to try it. Mainly just scraping product listings off a marketplace site, turned on my n8n, dragged in an HTTP node, a Cheerio node for parsing, a loop, a Google Sheets node at the end. All it took maybe 40 minutes and it worked first try which felt great.

Like the majority of the projects I've worked on, it then started throwing Cloudflare challenges after around 600 requests and that's where it stopped feeling great. I tried putting in some cheap datacenter proxies I had lying around from an old project, didn't help much, IP reputation on datacenter ranges is just garbage on anything halfway protected these days. Switched to a residential proxy pool instead and got further but still kept tripping something, which is when I remembered the IP is only half the story, the actual fingerprint matters just as much if not more. (take notes folks).

So I go to fix it in n8n and immediately went full stop, everyone who's done this before already knows about, which is that the visual nodes are amazing for the happy path and genuinely miserable the second you need anything custom. wanting to rotate user agents with actual entropy, not just a static list cycling in order. wanting real TLS fingerprint control so your handshake doesn't scream "I am a script" before you've even sent the request, wanting a headless browser session that actually behaves like a person scrolling and pausing instead of firing requests like a machine gun. none of that is a drag and drop node, you end up writing it in a Code node anyway which is just JavaScript wearing a costume, so you've reinvented half a script but now it lives inside someone else's execution engine and you can't easily version control it or run it locally without spinning up the whole n8n instance.

Compare that to just opening a .py file. requests or httpx if you want async, curl_cffi if the site's fingerprinting you (and these days almost everything past a certain traffic volume is), playwright if you actually need a full headless browser for JS rendered pages. yeah you're typing more in the first 20 minutes, but every single thing is yours, testable, debuggable with an actual setup trace instead of n8n's execution log that sometimes just says "error" and leaves you to guess. and when the scraper needs to scale, you're not paying per execution or fighting workflow timeout limits, you just run more processes or throw it on a queue.

At some point I just gave up taking care of proxy rotation and fingerprint config by hand and pointed the whole thing at a web scraper api instead, basically it felt a little like cheating at first but also I have a day job and the marginal value of me personally maintaining a TLS impersonation layer is zero. Aso tried doing a rough version of the same job in Go for comparison because why not, and that one was interesting for a totally different reason, the speed difference on concurrent requests was kind of stupid honestly, noticeably faster spinning up a thousand goroutines than even async Python, but the dev time to get there was longer and if you're not already comfortable with the language you'll burn an evening just on syntax instead of solving the actual scraping problem. so it's not really "Go is better" it's "Go is better if you already know Go and need raw throughput". One thing I didn't expect going in, mobile proxies actually outperformed residential on a couple of the trickier targets, something about carrier grade NAT making the IP reputation look cleaner since thousands of real phones share the same address anyway. didn't bother testing ISP proxies for this particular target since the site wasn't doing heavy ASN level scrutiny, but I've used them before on stuff where you want the static IP of a datacenter with the trust level of residential, good middle ground when rotation isn't what you need.

Then I poked at a search api for a side piece of this project, pulling SERP results instead of crawling category pages directly, and that ended up being way less of a headache than the rest of the whole ordeal combined, search engines have their own blocking logic obviously but it's a more solved problem than random ecommerce sites running custom bot detection.

What I keep landing on is it's basically a compromise between time to first result and ceiling. n8n wins time to first result by a mile. Python wins ceiling, not even close. Go wins ceiling even further out but only if speed at scale is actually your bottleneck and not, like, getting blocked every 400 requests or so regardless of how fast you can send them, which honestly is the actual bottleneck like 90% of the time, not raw speed.

Anyway I ended up keeping the n8n workflow for the parts that are basically just data movement, sheets, notifications, scheduling, and ripped the actual fetching logic out into a standalone Python script that n8n just calls and waits on. feels like the right choice.

Gotta hand it to my friend, while his solution was probably one of those in the million, this gave me a chance to try out different languages for scraping and whatnot.

Perhaps someone else found a way to make the no code route hold up against fingerprint based blocking, because every workaround I tried inside n8n itself felt like duct tape on duct tape and rinse and repeat.

TLDR: Spent a weekend comparing n8n scraping workflows against Python and Go for the same job, and n8n wins on speed to a working prototype but falls apart fast once a site starts fingerprinting you past basic IP checks. Tried datacenter proxies first (useless), then residential and even mobile proxies (mobile actually did better on a couple targets), but eventually just routed everything through a web scraper api since maintaining fingerprint evasion by hand wasn't worth my time. Python gave way more control for the actual scraping logic while Go only made sense if raw concurrent throughput was the bottleneck, which it usually wasn't compared to just getting blocked. Ended up keeping n8n for the boring data movement parts (sheets, notifications, scheduling) and pulled the real scraping into a standalone script it just calls.

I need an agent for BT UK. If anyone has one, please contact me.I'm using this for a group of Vinted registered stores. Could someone please help me?

Posted the latest Web Scraping Insider #8 if anyone here wants the full breakdown:

👉 https://thewebscrapinginsider.beehiiv.com/p/the-web-scraping-insider-8

https://preview.redd.it/073298wqhdah1.png?width=1200&format=png&auto=webp&s=fb13515fdeee641c3e79b23be01e364a5bfdb7d5

Quick summary of what's inside:

⚖️ When "Ethical" Proxies Aren't Ethical

"Ethically sourced" has become the proxy industry's favourite marketing word. Almost no provider will show you which apps their residential IPs actually come from - no public partner list, no audit trail, no independent verification.

The last couple of weeks made that gap impossible to ignore:

• Spur Intelligence scanned 6,038 LG webOS + Samsung Tizen apps - proxy SDKs in 2,058 of them (42.5% on LG, 26.9% on Samsung)
• Bright Data's SDK enrolling always-on smart TVs as exit nodes, with consent buried in TV remote arrow-key navigation
• SuperBox streaming boxes (sold at major US retailers) shipping with dormant Popanet proxy software - routing third-party traffic through home connections with no meaningful consent
• FBI/IC3 now warning consumers that everyday devices are being silently turned into proxy nodes

None of those device owners meaningfully opted in. Yet those same residential IPs feed pools sold as "ethical."

Our take: "ethical" should be a claim you have to prove - published partner list, audit trail, who consented / in which app / when - not a landing-page adjective. My bet is the market moves there within the next year or two.

---

🔮 Proxy Tester: now benchmarks residential proxies too (free for you)

We expanded the ScrapeOps Proxy Tester beyond proxy APIs. It already benchmarks ~15 proxy-API-style providers against your exact target URL. Now it does the same for residential pools, so you can compare both side-by-side.

https://preview.redd.it/bcaokzhthdah1.png?width=1163&format=png&auto=webp&s=e0abb2f36866657f4a0814bf0554d9c95093f661

How it works: submit your URL → real requests through each provider → every config they expose gets tested → ranked by success rate + cost per successful request.

Residential is where marketing fluff runs deepest ("30M+ IPs", "99% success rates"). From what we've seen across billions of requests, CPM rarely correlates with performance on your actual target.

Try it: https://scrapeops.io/proxy-providers/tester/

---

🥊 The browser wars are back: people are rewriting Chromium itself

For a decade, scraping browser innovation meant automation libraries on top of Chrome (Selenium → Puppeteer → Playwright). The browser underneath was treated as a commodity.

That may be shifting. Two forces:

1. Anti-bot reads deeper now - TLS, network stack, process behaviour - so runtime patches (playwright-stealth, undetected-chromedriver) break more often than they hold.
2. Chrome is heavy at scale. Thousands of concurrent browser instances (or long-running AI agents) make a purpose-built engine attractive on cost + startup time.

Projects worth watching:

• CloakBrowser - Chromium fingerprints patched at the C++ source level, not JS injection. Drop-in Playwright/Puppeteer replacement. Claims 30/30 on public bot-detection suites.
• Obscura - Rust headless engine from scratch, CDP-compatible so Playwright still talks to it. Claims ~70 MB binary, ~30 MB RAM, near-instant startup vs Chrome's 200 MB+ / ~2s. (Self-reported, v0.1.0 - treat as experimental.)
• Camoufox - modified Firefox with C++-level fingerprint spoofing. Strongest headless evasion in independent tests we've seen. Proves this isn't only a Chromium story.

Stealth is moving below the automation layer. Most of these are young and several lean on self-reported numbers - don't rip out your production stack overnight - but the direction is worth tracking.

Bottom line: the residential proxy supply chain is getting scrutinised from every angle (smart TVs, factory hardware, federal warnings), the browser layer is getting rebuilt from scratch, and the boring work still wins - benchmark on your targets, measure cost-per-validated-payload, not vendor adjectives.

Happy to discuss specifics here - especially if you've benchmarked

— Ian (ScrapeOps)

What is the best open-source scraping solution that you have used in the past year or so. Need a stable, working solution that I could run on my side. Preferably that same solution would support cloud integration.

Also should i use ISP or residential?

Mainly looking for something stable that reduces checkpoints and random suspensions. ive been using myprivateproxy + incognition setup and its been dog shit recently. Keep your answers simple please Im not an expert on this topic

HTTPX_RATE_LIMIT   = 1   # seconds between httpx requests
HTTPX_CONCURRENCY  = 3     # max parallel httpx fetches

please let me know if this is safe, and my IP won't be blocked. I am using the requests on company websites....

Hey all. Been picking up more automation contracts lately and Cloudflare keeps messing up the jobs mid-run. A few of my clients want competitor pricing scrapers, job board feeds, real estate data and almost every site worth scraping is sitting behind it now.

Rotating proxies used to handle most of this. Now runs are failing and I don't have a clean answer for clients beyond "Cloudflare got more aggressive" which isn't exactly inspiring confidence.

Trying to actually understand the full option rather than keep patching things after they break. What's holding up in production right now and what only works for a demo before dying two weeks later? Residential rotation, stealth browsers, managed scraping APIs, whatever.

Pricing transparency would also help since I need to factor infrastructure costs into client quotes before committing to a scope.

Started because I was overthinking this far too long and ended up reading about how free streaming apps on Samsung and LG TVs are literally enrolling your home IP into a residential proxy network while you watch Netflix or Prime on the other input. Like you click "agree" on some vague popup about "device resources" and your TV starts routing web traffic for whoever's paying for residential IPs that month. Up to 200GB or more, who knows?? All that while you're asleep. On your $2000 TV. Yall are fucking nuts if you think this is normal. And apparently that's just a thing that's happening and we should be all cool with it?? Boy whattahelly

Anyways, I went looking for who actually buys this traffic and why residential IPs are worth so much. And that's where it gets kinda funny. Turns out a huge chunk of the demand is scrapers trying to get past bot detection. Which, sure, makes sense, residential IP looks like a real human etc. Except I kept reading and finding that whole premise is like three years out of date. Cloudflare, Akamai, DataDome, the holy trinity. They're not primarily checking your IP anymore. They're fingerprinting the TLS handshake, the HTTP/2 frame ordering, your WebGL renderer string, how your canvas noise changes across sessions, whether your "mobile" session has gyroscope data. Your IP can be spotless and your Python requests library is still announcing itself on the first packet. Someone here on this sub literally tested this, datacenter IPs with proper TLS impersonation outperformed residential IPs with bare requests headers like 91% vs 60% on Cloudflare-protected site?? whaat?

So your TV is being used as a proxy node. The proxies are being sold to scrapers. The scrapers are getting detected anyway because the fingerprint gives them away. The TV thing was pointless to begin with basically. Am I looking to deep into this? Coz I feel like this shouldn't be normal, I mean the whole ordeal, not me looking into it. And then separately, completely disconnected from all this ordeal, some guy on Reddit figured out Spotify serves authenticated AAC files via CDN links that stay valid for 2 hours. Bro is going to get a DMCA notice fast if not already got it. What I'm trying to say is that Spotify already went through this exact thing. The tools are already built and already taken down. What else are we gonna have to deal with? And I'm talking about a small percentage of people that actually care, while the majority just clicks through all these popups without questioning a single thing.

Anyway I blocked proxyjs.brdtnet.com on my router, learned what JA3 fingerprinting is, and now I'm questioning every "agree" popup I've ever clicked. Normal tuesday, folks