u/Bharath0224

Hey all. Been picking up more automation contracts lately and Cloudflare keeps messing up the jobs mid-run. A few of my clients want competitor pricing scrapers, job board feeds, real estate data and almost every site worth scraping is sitting behind it now.

Rotating proxies used to handle most of this. Now runs are failing and I don't have a clean answer for clients beyond "Cloudflare got more aggressive" which isn't exactly inspiring confidence.

Trying to actually understand the full option rather than keep patching things after they break. What's holding up in production right now and what only works for a demo before dying two weeks later? Residential rotation, stealth browsers, managed scraping APIs, whatever.

Pricing transparency would also help since I need to factor infrastructure costs into client quotes before committing to a scope.

I feel like the market is pretty stale regarding web scrapers at the moment, sure there are variety of web scraping services all around the web, however, I feel like things got pretty stale. The whole early web was designed around human intent and Google figured that out early. Scraping services providers has spent years building infrastructure to extract data from systems that weren't meant to be extracted from. Now when you think about it, agentic search doesn't do that. What I mean is that It navigates the way a person would. What do you people think, does scraping will evolve into something agentic, or eventually get replaced by it? Coz I feel like something is going to change, but there is a lot of room for guesses.

It's me, ya psyop boy. I am 100% sure that this is going to upset some people and that's good coz this has been bothering me for quite some time and I'm done with the silence. Background on me: moderate scale scraping, 40-60k requests a day across a few different projects, been in this space about 3 and a half years. Started noticing something weird in my logs maybe last spring. Certain ASNs kept showing up no matter which provider I switched to. At first I just assumed that's how residential pools work, ISP coverage is what it is, whatever. Then I got obsessive about it and started auditing properly. I tested four providers. I am not naming them because I genuinely cannot afford a lawyer right now but if you've been doing this longer than six months you know exactly who I'm talking about. One is Israeli 🇮🇱🇮🇱 company (you already know), it markets itself extremely hard on LinkedIn and twitter and has a very american-sounding brand name, and one is the scrappy european one that keeps showing up in threads like this. The ASN overlap between two of them was 34%. Thirty four percent. Between the other two it was 29%. There is one specific /24 block that appeared in all four providers "proprietary residential networks." Same subnet. Four different dashboards with four different pricing pages all routing me through the same infrastructure. I also pulled whois on a bunch of these IPs and the rotation behavior is what really got me. Not just that the ASNs overlap, it's that the rotation patterns are identical. The TTL behavior. How they handle certain request headers. You don't get identical behavior from independent networks. You just don't. That's not how this works. My theory, and I want to be clear this is a theory and not me accusing anyone of anything specific, is that there's maybe three or four actual upstream aggregators who own the real infrastructure and basically everyone else is a white label on top of that. All those "95 million IP" claims in the marketing materials are all drawing from the same upstream pool. You are paying a premium to a middleman who bought access wholesale and is selling it to you retail with a nice dashboard slapped on top. What this actually means is that when you switch providers because your IPs are getting flagged too much, there's a real chance you are switching to the same IPs with different account credentials. You're not getting a fresh pool. You're just reintroducing yourself to the same exit nodes under a different invoice, huuuuh??? Exactly. I could be completely wrong about this and if someone from one of the big providers wants to come in here and show me actual evidence that their infrastructure is genuinely independent I will delete this post and take the L publicly...NAAWT. But nobody is going to do that because they all have NDAs that prevent them from explaining how anything actually works which is itself kind of telling.

Tell me you have a better idea of this whole ordeal, I dare ya

My two cents. Most people spend bunch of time reverse engineering request headers when the browser will just hand them to you. Next time you find an API call in the Network tab, right click it and hit Copy as cURL. Paste it into your terminal and it works instantly, cookies, headers and all. From there you can import it directly into Postman or use a tool like curlconverter to turn it into clean Python requests code in seconds. The browser already did the hard work of figuring out what the server needs. There's no reason to reconstruct that by hand

Hi there! I'd like to understand the difference between companies showing up as "Business" status versus "Search Term." I've also seen that Smart Keywords are available, do they play a role in changing a term's status to Business? What are these Smart Keywords. Could you provide a quick overview and some practical tips on how to achieve Business status? Additionally, does having Business status indicate that the term is being searched more frequently on Reddit?