u/ahiqshb

A few weeks back I set up kind of a trap on one of my sites (side projects), I hid a link to a fake git repo inside an html comment on a page that no one would probably ever read, and that repo had some bogus endpoints in a shell script. nothing linked to it anywhere else, no sitemap, nothing. Now, fast forward almost three weeks and I start seeing hits on those exact fake endpoints coming from Amazon IPs, (I checked the official amazonbot ip range list) and yep, what do you think? IT MATCHED THEM. The thing is my robots.txt explicitly disallows amazonbot, has for a while. So either they don't check it or don't care. What really got me is it wasn't using the amazonbot user agent when it hit those endpoints, just to what looked to me was generic browser string. So it's not even pretending to follow the rules, it's actively hiding what it is while going after stuff hidden in source code that a normal crawler would have no business finding. Also weirdly it didn't touch links inside the fake markdown docs I had set up there, only pulled the url out of the shell script, and it waited a full day before trying it, then tried again a day after that. Maybe it was like parsing code for training data on a delay? I am not too sure what their take was. I've checked the net for similar situations and it's not illegal to do so, but I just wanted to vent because I always assumed the big names were at least somewhat well behaved compared to the sketchy scrapers, apparently not.

TLDR: Amazonbot ignored my robots.txt and used a fake user agent to hit an endpoint that was never linked anywhere

A few weeks back I set up kind of a trap on one of my sites (side projects), I hid a link to a fake git repo inside an html comment on a page that no one would probably ever read, and that repo had some bogus endpoints in a shell script. nothing linked to it anywhere else, no sitemap, nothing. Now, fast forward almost three weeks and I start seeing hits on those exact fake endpoints coming from Amazon IPs, (I checked the official amazonbot ip range list) and yep, what do you think? IT MATCHED THEM. The thing is my robots.txt explicitly disallows amazonbot, has for a while. So either they don't check it or don't care. What really got me is it wasn't using the amazonbot user agent when it hit those endpoints, just to what looked to me was generic browser string. So it's not even pretending to follow the rules, it's actively hiding what it is while going after stuff hidden in source code that a normal crawler would have no business finding. Also weirdly it didn't touch links inside the fake markdown docs I had set up there, only pulled the url out of the shell script, and it waited a full day before trying it, then tried again a day after that. Maybe it was like parsing code for training data on a delay? I am not too sure what their take was. I've checked the net for similar situations and it's not illegal to do so, but I just wanted to vent because I always assumed the big names were at least somewhat well behaved compared to the sketchy scrapers, apparently not.

TLDR: Amazonbot ignored my robots.txt and used a fake user agent to hit an endpoint that was never linked anywhere

Quick update for anyone still tracking this. After the FBI/IRS-CI seizure back in July and the whole Popa botnet mess, netnut.com and netnut.io are still behind the seizure banner, but Alarum just quietly stood up a new site at netnut.ai. New branding, new positioning too, it's leaning hard into the "AI era" now with an LLM scraper API and RAG/agent framing instead of the old residential proxy messaging, (they still sell those, just not being pushy about it.)

No public statement yet about the Popa allegations or what changed on the sourcing side. But still, the company took a hit and lost thousands of clients because of this mess. Will update if Alarum puts out anything official

Firecrawl split their doc parsing into two separate repos, pdf-inspector for PDFs and AnyDoc for basically everything else (docx, xlsx, pptx, epub, csv, the greatest hits list of 14 formats total).

From what I gathered, instead of throwing every PDF through OCR, it looks at the internal structure first (fonts, text operators, that kind of thing) and figures out per page whether it's text or needs OCR. They're claiming a decent speed bump on mixed docs because of it.

AnyDoc is another thing, single Rust binary, no API key, no external deps, just one function call and you get markdown out. They ran it against the usual suspects (LibreOffice etc) on 94 docs and it came out ahead on coverage and speed by a pretty wide margin. Yes, I know this may sound exaggerated by them, but since it's just been out, we will have to wait for some independent testers to verify all this. Anyways, both are already live under the hood of Firecrawl's /parse and /scrape endpoints, but the repos are standalone. Also, Rust only for now apparently.

Perhaps you guys heard it too and would like to share something additionally?

Spoiler: it's not the model. It's not RAG. It's a markdown file.

Been some time since I shared about the builds so here it is. I built a deep research agent that takes open-ended questions, runs 15-30 tool calls over 10-25 minutes, and produces structured reports with sources. The model choice mattered less than I expected because the structure around it is where it's at.

Multi-model flow beats single-models. LangChain's open_deep_research repo (12k stars) does this well. They use gpt-4.1-mini for summarizing search results, gpt-4.1 for research reasoning, and a separate model for compressing accumulated findings before the context window blows up. This separation alone outperforms a single frontier model at everything. Their benchmarks on Deep Research Bench (100 PhD-level tasks, 22 fields): GPT-5 scores 0.49, default gpt-4.1 config scores 0.43 at ~$46 total, Claude Sonnet 4 hits 0.44 at $187. Do the math.

External state management is the savior. LLMs lose coherence after ~5 tool calls in a long-running sequence. Simplest implementation looks like this: the agent writes a markdown todo file at the start, updates it after each step, and re-reads it before deciding what to do next. This gives the model explicit visibility into what's done, what's in progress, and what's remaining. Without it, agents routinely skip subtasks or repeat work. Neat addition? I know :D Then LangGraph solves this at the structure level with built-in state machines and checkpointing. The todo file approach is more transparent and debuggable. LangGraph's approach seems to scale better.

My suggestion - don't dump full search results into context. Here's how I do it. Search → summarize to 2-3 sentences → store summary → proceed further. Pull full content only during synthesis. This dropped my token usage ~50% and improved output quality because the model wasn't being overwhelmed in irrelevant text. (Again, the token usage, not sure how accurately the LLMs give this data, but from my calculations it was around 50% In general, a typical deep research session consumes 50-150K input tokens and 10-30K output tokens.

Another thing to mind, LangSmith tracing. With it I could see the agent calling the same search query three times consecutively. You cannot debug agentic systems from outputs alone. You need the full tool implementation sequence, reasoning chain, and token counts per step.

My current setup is this: Claude Sonnet 4 for reasoning, gpt-4.1-mini for summarization, Tavily for search, you can also use Oxylabs Fast Search API for faster response times and you get structured results quickly. Choose what;'s better for you. Filesystem backend persisting intermediate findings as JSON. Runs as a LangGraph state machine. $2-4 per report.

Hope you'll find this interesting and useful.

It's no secret that structured output quality varies widely across LLM scraping targets. And for those who are using LLMs on the daily, it has been very noticeable. For example, ChatGPT's Markdown formatting stays fairly stable, Gemini's moves more, and Google keeps adjusting how citations come back. Also, I noticed that oxylabs rolled out a dedicated Gemini source in their Web Scraper API, anyone tried it already? Would be nice to know about this source more and what it does from the first hand experiences.

Ok so this is gonna be a bit of a ramble but I need y'all to hear this. I work adjacent to the scraping/data space (not naming employer, y'all know how it is) and I always kind of just you know, accepted that "residential proxies" were this slightly grey but mostly fine thing. Like yeah obviously somebody's home IP is being used, somebody agreed to something somewhere, moving on. Then yesterday I see netnut is just gone. Not down, not maintenance page, straight up FBI DOJ IRS-CI seizure banner, Google and Lumen and Shadowserver all stamped on it too like it's a whole coalition operation. Like bruh, I have never in my life seen this on any website. So imagine the shocker lol. And I'm sitting there like wait since when did the IRS care about proxy IPs??? Turns out IRS-CI does financial crime investigations and it could be related to that, but still, seeing that logo on a proxy provider's main website??? Diabolical mate.

So I go searching what's going on and apparently it's tied to this thing called the Popa botnet, basically it's been running for like four years hijacking Android TV boxes (remember that Bright Data SDK thingy?? And Krebs traced a chunk of it back to actual NetNut infrastructure through some ex-employee's domain. Not some randoms, an actual former VP of R&D there. Google's own blog post says they think the network was over 2 million devices at one point and that a lot of those "different" proxy brands you see around are just NetNut wearing a different logo through resellers. I swear I seen a post talking about how there is one or two proxy providers sharing those IPs/reselling whatever. Makes sense now, people?? Am I the only one concerned here with a few other nerds? And to be fair I want to be fair here, Alarum (the parent company) came out and started calling it that a botnet is inaccurate, also said that there's real KYC and consent flows and misuse detection on their end. So it's not like this is some minor thing everyone agrees on, it's actively disputed and I think that matters, I'm not trying to just slam a company because Krebs wrote a scary headline, no. Make what you will out of it, but I things are rising to the surface, more and more often. Again it does make me think about how much of the "residential proxy pool" any of us are touching day to day is actually consented in a way a normal person would recognize as consent, versus consented in the "buried in paragraph 14 of an SDK terms screen" way. Like where's the line between legit residential network and just a nicer branded botnet, and how would any of us even know the difference from the outside.

Not trying to start a witch hunt on the whole industry, genuinely just spent my evening reading legal filings and threat intel blogs instead of sleeping like a normal person, but after Bright Data SDK shennanigans and now Netnut, boy, there's gonna be more stuff coming out. So I figured I'd share since I know a bunch of you here actually work with this infra daily and probably have opinions on these matters.

Sources if anyone wants to go check themselves.

Krebs on Security, the original Popa botnet reporting: https://krebsonsecurity.com/2026/06/popa-botnet-linked-to-publicly-traded-israeli-firm/

Google's threat intel writeup on the disruption: https://cloud.google.com/blog/topics/threat-intelligence/google-continued-disruption-residential-proxy-networks

Alarum's official response: https://alarum.io/alarum-technologies-responds-to-inquiry-into-residential-proxy-networks/

And divinetworks.com itself, which is also showing the seizure page now: https://divinetworks.com/

P.S Some time later I found that they mixed up the domains because .com is down, but not .io which is the main page of netnut.

If anyone's dealt with Hcaptcha while web crawling or running bots? I'm looking for something that can solve this without running a full Chromium/Firefox instance. The RAM and CPU overhead is not worth it for what I'm doing.

Ideally something that takes the sitekey and current URL and handles it without me having to babysit a whole browser session. I've seen a few repos floating around but most of them are either abandoned or don't work reliably anymore.

Anyone have something they actually use in production or at least tested recently? Would appreciate any help

This is a newbie friendly thread. Feel free to contribute if you think that your input will help the newcomers.

What mistakes did you do when using proxies for the first time? For Scraping? What adjustments you thought were worth the time. How is your approach differs from when you started to use the proxies?

Seeing how the TikTok fingerprinting post that blew up recently finally pushed me to write something I've been sitting on for a while too. Quite a few people DM'd me asking about my stack throughout several months, so here it is. Full breakdown, no provider shilling, honest and real discussion are welcome. (will make some spaces for easier reading).

That DesperateCoyote post nailed something most people in this space dance around: your IP being "clean" is basically table stakes now, not a competitive advantage. Platforms don't primarily care about your IP anymore. They care about whether your device behaves like a real device. I ran my own tests over few months across TikTok, LinkedIn, and a few e-commerce targets and the pattern was consistent. Sessions from "your beloved clean" residential IPs but running on a standard headless Chrome setup got flagged within 15 to 30 minutes on average. Sessions from a datacenter IP but running through Velum with proper sensor injection? Significantly longer survival rates. That's not a one-off result, thats the signal. The reason platforms moved this direction is actually pretty logical once you think about it from their engineering side. IP reputation databases are commoditized. Every half-decent anti-detect tool cycles through residential IPs so the defense shifted to the hardware layer, because that is harder to fake at scale.

The DesperateCoyote post described the fingerprint cluster score as a weighted confidence interval on "real human on a real device" which is accurate but undersells how granular it actually gets. WebGL renderer string consistency is one of the bigger ones, and not just that you have one but whether its consistent across sessions for the same "device." If your renderer string changes between sessions thats an immediate mark against you. Velum handles this quite well out of the box if you lock your profile, but it requires you to actually manage profile storage properly rather than spinning up fresh contexts every single run. Touch event timing is where things get genuinely impressive on the platform side. Real thumbs are sloppy, they drift, pressure varies, tap-to-tap timing has natural variance. Synthetic touch events are clean in a way that human fingers never are and the fix isnt randomizing your timing, its using distributions that actually match human motor patterns. This is where 8n8 has been doing interesting work, specifically the sensor emulation layer in their mobile profile configs. Battery API polling behavior is one I hadn't thought about much until this year. Real devices have battery levels that change over time in predictable ways, so a session that starts at 78% and is still at 78% four hours later is suspicious. Small signal but it adds to the cluster score. The timezone, DNS resolver, and behavioral gap is probably where most people get caught without realizing it. You can have a perfect German residential IP and a German timezone in your headers and still get flagged because your DNS resolver is resolving through a US-based server, or because your behavioral patterns like typing speed, scroll velocity, time-on-page don't match the demographic profile of a German user. Prismatic addresses this with what they call geo-behavioral profiles which bundle together not just the network layer stuff but the user behavior simulation parameters too. More holistic approach than anything else I'd seen and to be fair, it changed how I thought about the whole problem.

A lot of people in this space build scrapers and then bolt on some kind of orchestration layer as an afterthought. I was guilty of this for way too long, had scrapers running on cron jobs with zero retry logic, no rate management, and no session lifecycle management. Switching to Make for orchestration changed the output quality more than any proxy upgrade I made that year. When you manage workflows properly through something like Make you naturally end up with human-paced request timing because the automation has overhead between steps, session reuse instead of constant fresh session spawning which is itself a detection signal, proper error handling that doesn't hammer endpoints when things break, and actual audit trails so you can diagnose when and why sessions get flagged. The pacing thing sounds obvious but most people building direct scrapers optimize for speed and then add sleep() calls as an afterthought. Thats not the same as naturally paced requests coming from an orchestration layer doing real work between steps. Make's webhook and scheduling infrastructure also makes it way easier to implement robots.txt compliance and proper crawl-delay respect, which brings me to something I want to push back on slightly.

There was another post written on this sub that talks about big tech hypocrisy, and it is emotionally satisfying and honestly not wrong on the facts. OpenAI, Google, Meta et al absolutely did build on scraped data without permission and are now going after people for doing the exact same thing. The "fair use for innovation" asymmetry is worth being angry about. But I think it creates a mental model that's counterproductive for people actually building in this space. Whether scraping is ethical is a less interesting question than what the actual legal and technical constraints are and how to work within them. robots.txt is not legally binding in most jurisdictions but violating it is evidence of bad faith. The hiQ v. LinkedIn ruling established that scraping publicly available data generally isn't a CFAA violation, but courts look at whether you circumvented access controls and robots.txt plus explicit blocking counts toward that. Respecting crawl delays isn't just polite, its actually relevant to your legal exposure. ToS violations are a civil matter in most cases, not criminal. The Computer Fraud and Abuse Act has been interpreted narrowly by most courts post-Van Buren so violating a terms of service alone without credential-based access doesn't typically rise to unauthorized access under the CFAA. And even if you're legally clear, hammering a small site's server is just bad behavior. The Brendan Martin tutorial made this point well, one request per page, save locally, parse separately. Should be baseline for everyone and it usually isn't.

I am using several providers, as you would already guessed, however I will not say specific provider names because the landscape changes fast enough that recs go stale quickly, but the stack architecture I'm running now is ARM-based cloud phone environments for mobile targets (DesperateCoyote is right, this matters more than the proxy layer), Velum for profile management and sensor injection on desktop targets, 8n8 for mobile sensor emulation where Velum doesn't cover it, Prismatic for geo-behavioral profile bundling on targets where demographic coherence actually matters, Make for orchestration and scheduling, and local storage of raw HTML before parsing because yes, always, still. The shadowban rate on TikTok-adjacent targets dropped from roughly 55% to under 10% after moving to ARM environments. The proxy change alone, same environment just better proxies, moved it maybe 8 percentage points. The environment change moved it 37 points. That pretty much tells you where the leverage is.

There's a few things I haven't fully figured out and genuinely want input on. Canvas noise variance patterns specifically, DesperateCoyote mentioned they track how your noise changes not just that it exists, and I haven't found a clean way to implement dynamic noise variance that feels natural across sessions. Has anyone actually cracked this in a way that survives more than a few days? The accelerometer absence signal on mobile is another one. If you're running ARM cloud phones you have real accelerometers which is the obvious fix, but for desktop sessions claiming to be mobile is there a better approach than just not claiming to be mobile? Seems like the cleanest answer is just use the right tool for the target but I'm curious if anyone's found a workaround. Also been wondering about Make vs n8n for scraping orchestration specifically. I've been on Make for a while but n8n's self-hosted option is increasingly appealing for cost reasons on high-volume jobs. Anyone switched and have thoughts on the practical differences for this use case? And Prismatic's geo-behavioral profiles on non-Western targets, their coverage seems strong for US and EU but has anyone tested on Southeast Asian or LATAM targets where user behavior patterns are going to look pretty different? If I'm wrong about something say so and explain why, that's way more useful than a downvote.

I've been following the OpenAI lawsuits and the one detail I can't stop thinking about: a 19-year-old asked ChatGPT about mixing sedatives, it acknowledged the combo "could be risky", then gave him dosages anyway, added Benadryl to the recommendation, and told him to go lie in a dark room instead of seeking help. He died. Source. The Canadian case is somehow worse. OpenAI's own safety team flagged the shooter's account for "gun violence activity and planning" months before the attack and pushed to notify authorities. Management said no. Source. At some point "we're just a general-purpose tool" stops being a defense. Where that point is, that's what these trials are actually going to decide. Guardrails are coming whether the industry wants them or not. Every lawsuit forces a paper trail. And when harmful outputs become liability, the instinct is aggressive filtering, mandatory escalation triggers, activity logging with retention policies. Fine for consumer chat, however, for more tech enthusiasts its going to be brutal. Now the real risk for scraping and agentic workflows is over-correction. If "how do I access this data at scale" gets flagged the same way "how do I build a weapon" does, open-weights models win by default. It would make me want to just run it locally and skip the compliance layer entirely. The smarter play would be tiered access, stricter defaults for consumer products, more permissive behavior for verified API users with actual business context, but that requires product nuance, and right now OpenAI is in legal defense mode.

My bet is that we should expect more API friction over the next 12-18 months. Local models are about to get a lot more interesting.