r/ProxyEngineering

Hi, I’m fairly new to this community and trying to understand proxy and bot-management behavior in a bit more depth.

I ran into a strange case recently. The same browser/client setup works fine over a direct connection, but as soon as I enable a proxy, Cloudflare returns an immediate 403 on the first request.

I tested a fresh session, an already working session before enabling the proxy, and a few different proxy exits. Same result every time it works directly without a proxy but the proxy gets blocked immediately.

Because it happens on the first request, I don’t think rate limiting or an old/bad session explains it.

My first thought was IP/ASN reputation, but I’m curious what others would check next. Would you mainly look at the proxy/network reputation, geolocation, or something else changing when the proxy is introduced?

I’m mainly trying to understand how people would diagnose this rather than just keep switching proxies until one works.

Hey everyone,

I'm Ian Kerins, CEO & Co-Founder of ScrapeOps.io.

After four great AMAs with the r/WebScrapingInsider community, we're excited to bring you our fifth guest.

This Thursday, August 20, at 10:30 AM GMT+3, we'll be joined by Stan Sadokov from NodeMaven for a discussion around one of the questions almost every serious scraping operation eventually has to deal with:

How do you actually choose the right residential proxy provider?

NodeMaven is a proxy infrastructure provider focused on residential, mobile, and ISP IPs, with an emphasis on IP quality rather than simply advertising the biggest pool.

Instead of relying purely on massive raw IP counts, NodeMaven uses real-time quality filtering to identify and remove flagged or low-reputation IPs before they cause problems for customers.

WebScrapingInsider AMA #5 with NodeMaven

During the AMA, we can dig into topics like:

• Residential proxy quality
• How proxy providers measure IP reputation
• Choosing between residential, mobile and ISP proxies
• IP rotation vs. sticky sessions
• Proxy success rates and why they vary by target
• Detecting and removing bad IPs
• Proxy performance for large-scale web scraping
• How proxy pricing actually works
• What to look for when evaluating a proxy provider
• Where residential proxy infrastructure is heading

We've had some great discussions in the community so far:

Our first AMA covered proxy infrastructure, Cloudflare bypass, browser automation, and scaling scrapers.

Our second with WebClaw explored AI agents, hidden APIs, open source scraping, and LLMs.

Our third with CloakBrowser went deep on stealth Chromium, fingerprinting, anti-bot detection, and browser automation.

Our latest with Browser Use brought insights on browser agents, AI-powered scraping, proxies, evaluations, and browser infrastructure.

We're excited to keep the conversation going.

If you're building web scrapers, data pipelines, browser automation, account management systems, or proxy infrastructure, or you're simply trying to figure out why one proxy provider works better than another, this should be a good one.

Drop your questions below, and Stan Sadokov from NodeMaven and I will start answering them during the AMA.

Looking forward to seeing everyone there!

Ian

Can any expert guide and suggest me proxy for trustpilot review management, or any one have this same expirenced who guide me as well.any expertise need your attention here.

I'm looking for something with AI capabilities built in, like Tavily or Exa. Main use case is automating research and scraping workflows as part of agentic pipelines.

I saw the other post where people recommended residential proxies, please be understandable and provide your take, examples and opinion, not just straight up promo bs

[removed]

i'm setting up a few cloud phones for social media I handled and trying to figure out what proxy type makes the most sense. I've seen people use residential and mobile proxies but i'm not sure if one is actually better for this use case. Mainly looking at stability and consistent speeds, not really trying to find a specific provider. What are you guys using with cloud phones?

Running a rank tracker for my side project, need localised SERPs across few US cities. Around 3-4 queries a day.

Datacenter died in the first 50 queries. Residential held for maybe 200-300 then trust tanked and everything started rate-limiting. Started to think about high quality mobile proxies.

Stack right now:

- Python with curl_cffi, impersonate="chrome" for JA4
- 3-6s jitter between requests, no fixed intervals
- gl / hl / uule params for locale
- Response body scanned for CAPTCHA marker so I rotate before wasting queries

Scraping side feels fine, the network layer is what's killing me.

Comparing NodeMaven, Voidmob and DataImpulse. Not going the managed SERP API route, at my volume that's $8-10k a month.

What I care about:

- Real 4G/5G carrier IPs (verified via IPQS)
- Sticky sessions for repeated queries from same city
- Per-request rotation for broad keyword sweeps
- City-level geo-accuracy, not just country

Anyone doing SERP scraping at similar volume, what mobile pool are you actually holding trust on?

Thanks!

Back in the day, I used VPNs to manage multiple accounts. It looked simple but painful. CAPTCHAs, random logouts, verification checks, and accounts suddenly looking suspicious.

Where VPNs start to break

- One IP for multiple accounts: VPN changes the connection for the whole device. Five accounts can still share the same exit IP and network identity.

- One location change affects everything: Switch servers and every account suddenly “moves” at the same time.

- Limited IP control: You usually choose a server, not a stable IP assigned to one specific account.

- No network fingerprint isolation: Different IPs don’t automatically mean different browser or device identities.

Why proxies make more sense

With proxies, each browser profile or account can have its own connection.

Account A → Proxy A
Account B → Proxy B
Account C → Proxy C

Mobile proxies also use carrier networks, which can better match normal mobile traffic.

The right setup

A proxy is only one layer. Proper separation usually also means:

• isolated browser profiles
• antidetect browsers
• cloud phones
• consistent timezone/location
• controlled IP rotation

Proxy = network separation.
Browser/cloud phone = device separation.

That difference would have saved me a lot of CAPTCHA-solving years ago.

Ever notice how some "residential" proxies seem to perform worse than they did a few months ago, even when you're buying from the same kind of provider? I started looking into this after burning through way more proxies than I expected and eventually found that a lot of what gets sold as residential can still show up as hosting or datacenter infrastructure.
That might not matter much for basic scraping, but once you're dealing with stricter platforms, the difference becomes pretty noticeable. One thing I started checking more closely was the ASN/ISP info. If the IP is showing up under a hosting provider instead of the ISP you'd normally expect from a residential connection, that's usually a pretty good sign that the pool isn't what I thought I was paying for.
I've been testing proxyshard recently and, at least in my case, their residential pool has been pretty consistent. I've been running my current volume for about a month now and haven't seen the same drop-offs I was getting with some of the other pools I tried. Obviously that's just my experience so far and things can change over time.
The only real annoyance I've had is support being a little slow on non-urgent questions. Nothing terrible, but it does mean I've had to rely on their docs more than I'd like.
So yeah, these days I don't really take the "residential" label at face value. I usually check the ISP/ASN details first and make sure the IPs actually look consistent with what I'm paying for. It's saved me a lot of wasted time and money compared with just assuming the provider's label tells the whole story.

Hey, basically the title, I am looking for another Residential Proxies provider as the current one started having issues with the infra, and my projects can't stay idle while they fix their issues. Anything trustworthy to recommend?

A few weeks back I set up kind of a trap on one of my sites (side projects), I hid a link to a fake git repo inside an html comment on a page that no one would probably ever read, and that repo had some bogus endpoints in a shell script. nothing linked to it anywhere else, no sitemap, nothing. Now, fast forward almost three weeks and I start seeing hits on those exact fake endpoints coming from Amazon IPs, (I checked the official amazonbot ip range list) and yep, what do you think? IT MATCHED THEM. The thing is my robots.txt explicitly disallows amazonbot, has for a while. So either they don't check it or don't care. What really got me is it wasn't using the amazonbot user agent when it hit those endpoints, just to what looked to me was generic browser string. So it's not even pretending to follow the rules, it's actively hiding what it is while going after stuff hidden in source code that a normal crawler would have no business finding. Also weirdly it didn't touch links inside the fake markdown docs I had set up there, only pulled the url out of the shell script, and it waited a full day before trying it, then tried again a day after that. Maybe it was like parsing code for training data on a delay? I am not too sure what their take was. I've checked the net for similar situations and it's not illegal to do so, but I just wanted to vent because I always assumed the big names were at least somewhat well behaved compared to the sketchy scrapers, apparently not.

TLDR: Amazonbot ignored my robots.txt and used a fake user agent to hit an endpoint that was never linked anywhere

I was a bit sceptical when I first read this post last week, I was honestly thinking it is a copycat website.

But - netnut has officially updated their new netnut.ai website on their official linkedin profile as well and it was confirmed verified page by linkedin 12 days ago

Screenshot:

https://preview.redd.it/y2m1cri9nxjh1.png?width=796&format=png&auto=webp&s=28b587dbaa61008658293d93c05da49c6dd1d17f

They've moved away from presenting themselves primarily as a residential proxy provider and are now positioning the company as:

• The data layer for AI
• The web data infrastructure for the AI era

When your previous domains have been seized and the industry has spent weeks questioning how your network was sourced, simply changing the story isn't enough. Customers buying infrastructure care about reliability, but they also care about transparency and confidence in the company behind it and I don't believe in this company anymore?

Companies like Bright Data (they had their own sourcing problems as well which everyone knows about), Oxylabs Nodemaven, decodo and others have all expanded beyond proxies into scraping APIs, datasets and AI-focused products. The market is slowly moving away from selling IPs and toward selling access to web data

So I guess the cycle is

1. Proxy company makes a mistake of sourcing IPs and blatantly using their customers data

2. Gets seized by FBI

3. Buys new domain

4. Gets repositioned from proxy provider to "DATA LAYER FOR AI"

Like what even is this?

How is this supposed to work this way?

A completely banned company, just used the same logo, the same mascot, the same colors, just slighlty different messaging

ANd it's back online?

Damn

I've been trying to scrape public data using rotating datacenter proxies, but Cloudflare/Akamai keeps flagging them almost instantly.

Is anyone still successfully using Datacenter proxies for scraping, or is switching to Mobile/Residential proxies mandatory now? Would love to hear how you guys handle IP reputation and fingerprinting

Quick update for anyone still tracking this. After the FBI/IRS-CI seizure back in July and the whole Popa botnet mess, netnut.com and netnut.io are still behind the seizure banner, but Alarum just quietly stood up a new site at netnut.ai. New branding, new positioning too, it's leaning hard into the "AI era" now with an LLM scraper API and RAG/agent framing instead of the old residential proxy messaging, (they still sell those, just not being pushy about it.)

No public statement yet about the Popa allegations or what changed on the sourcing side. But still, the company took a hit and lost thousands of clients because of this mess. Will update if Alarum puts out anything official

I have been going back and forth on this for the full scraping setup and found something useful. PycURL's multi-handle interface would seem like it is still the way to go if you're doing huge numbers of requests in a minute and need libcurl level control (custom SSL certs, connection reuse tuning, no async runtime dependency). It's a bit wack to set up on macOS though so keep in mind. I think that httpx is the better call once you're doing async work. AsyncClient with gather() on a shared proxy handled 10 concurrent requests without the boiler plate PycURL needs for the same throughput, and HTTP/2 comes free. Rotation without health checks seems to eat up IPs for some reason, sticky sessions per proxy for login/form flows works better for rotation.

I’m scraping a betting site and considering IP rotation to deal with their CloudFront 403s. I’ve tried different request rates, jitter, headers/TLS, etc., and I never go above \~20 requests/minute. I’ve also tried figuring out their rate/volume limits or any consistent pattern behind the blocks, but I haven’t found anything clearly defined. For now, rotating IPs is what has worked best for me — distributing the requests across different IPs lets me scrape for much longer, and I’m not currently running into blocking issues.

I’m not doing any login/account stuff or accessing anything private; I’m only collecting publicly available betting odds/data.

I know IP rotation is extremely common in scraping, but I’m not sure if it could be problematic in this specific context. Could rotating IPs to avoid CloudFront restrictions get me into legal trouble, or is it generally fine? I’m aware the site may not like it, but I’m mainly wondering whether there’s a legal issue with doing it this way.

lol ok, so back in june i was setting up something that needed webrtc + fast udp, went proxy shopping like normal. every provider's landing page has "full socks5 support!!" plastered everywhere so i figured cool, socks5 = udp included, thats like the whole point of socks5 over socks4 right
wrong. bought 3 different packages from actual known brands (not naming them, dont wanna deal with that) and every single one either timed out or quietly fell back to tcp. spent way too long assuming it was my own code before i even considered it might be the proxy
finally got someone on live chat after like 20 min of hold music and turns out yeah, "socks5" on their plan is tcp only, udp is some separate enterprise thing that costs an extra arm and a leg a month. another provider's rep just kept trying to upsell me instead of answering the actual question lol
anyway apparently this is just a thing? providers advertise socks5 and quietly mean the tcp-only version because most people buying it never touch udp anyway so nobody notices until you actually need it for something real
switched to proxyshard about a month ago, this one actually has udp working out of the box no upsell needed, and everything just works now. still annoyed i burned 3 different subscriptions figuring that out though
moral of the story i guess, dont trust the badge on the site, just ask support point blank if THIS exact plan supports udp before you pay. would've saved me a month and like $150 honestly

The tricky thing with screen scraping is that most people reach for it when they don't need it.

The difference

Web scraping reads a page's HTML or an API. The data arrives already structured, tags and values you can grab directly. Screen scraping reads what's rendered on screen, so the data arrives as pixels and you need OCR to turn it back into text.

https://preview.redd.it/ovo6pbl07zih1.png?width=903&format=png&auto=webp&s=9fe453509e0405de1b4051afc1392b7d5a4458fe

Why the extra step matters

More stages means more places for things to break, and OCR errors are sneaky. In a quick test with Tesseract, clean text read at 100% accuracy, but a slightly low-res capture hit 96% and still flipped a price from 1299.00 to 2299.00. A blurry one turned a 4.6 rating into 46. The headline accuracy looks fine while single digits quietly corrupt your dataset.

When to actually use it

If the data exists in the HTML or an API, parse that. Screen scraping is for when it genuinely only exists visually, like prices rendered as images, embedded charts, scanned PDFs, or legacy terminal systems with no API at all.

If you do go the OCR route

Capture tight regions instead of full pages. If the number you need sits in one panel, screenshot that panel. Less noise means cleaner OCR output, and small layout changes elsewhere won't break your job.

https://preview.redd.it/m7zo8tj17zih1.png?width=970&format=png&auto=webp&s=08c196ae71b80971a0fc1368691f3b9ec11c6337

Validate output against expected formats, especially numbers. Don't trust raw OCR text for anything that ends up in a dataset.

For JS-heavy pages, render in a headless browser first, then capture. A plain HTTP request to a single-page app returns an empty shell.

One last thing

Budget for maintenance. A code-level parser breaks when the markup changes, a screen scraper breaks when the layout changes, and layouts change more often.

Curious if anyone here has run OCR pipelines at scale, and how you handle validation for numeric fields.

One thing i have started paying much more attention to with scraping is the history of the ip itself.

I used to compare proxies mostly by speed, price and location. If the requests were fast and the proxy worked, I assumed i was good.

Then i started noticing that two proxies from the same location could behave completely differently.

One would work normally while another would start getting challenges, throttling or weird responses almost immediately.

That's when I started looking at ip reputation as a separate thing from proxy quality.

For me, the basic process now is something like this:

1. Test a batch of ip's before putting them into a larger job
2. Check how the target responds to each ip instead of assuming every ip in the pool is equal
3. Keep track of response codes, challenge rates, connection errors and latency over time
4. Remove consistently problematic ip's instead of repeatedly retrying them
5. Avoid hammering a single ip just because it's working
6. Keep request rates reasonable and cache anything that doesn't need to be fetched again

The interesting part is that a slower ip with a clean history can sometimes be much more useful than a faster ip that constantly triggers additional checks.

I also stopped treating "X million ip's in the pool" as a particularly useful metric by itself. If a large percentage of those IPs aren't suitable for the websites you're actually working with, the headline number doesn't mean much.

Obviously this isn't a way around every anti bot system. some sites are going to look at much more than the ip and changing proxies won't fix a bad crawler implementation.

Curious how other people measure ip quality in their own scraping setups. Do you actually keep reputation or success rate data per ip, or do you just let the proxy provider handle the pool?