what worked a year ago feels like it's falling off fast, so curious what people are actually running now.

for the sites that really fight back, cloudflare, datadome, the nastier akamai stuff, are you just going straight to a browser now or still trying http first with curl_cffi and only spinning up a browser when you have to? and honestly i can never decide how much of it is proxy quality vs just having your client set up right. feels like i blame the wrong one half the time.

not fishing for anyone's secret setup or anything, just trying to figure out if there's some rough consensus on what a sane setup looks like in 2026 or if everyones still just duct taping their own thing per site.

The Akamai solver supports V2 & V3 sensors and the pixel challenge.

It doesn’t use a browser, however it runs a minimal V8 sandbox.

NodeJS: https://proofofbots.github.io/web-re-toolkit/packages/node/

Python: https://proofofbots.github.io/web-re-toolkit/packages/python/

It is a massive monorepo so just to clarify: The web-re-toolkit itself is a project that provides tools for web reverse engineering, Akamai (and other) solvers are just a small part that use it.

The NPM & PyPI packages are just wrappers for the client built with WRE (github.com/proofofbots/web-re-toolkit) so they include a binary which contains the actual sandbox and solver. All of it is open sourced, so feel free to compile it yourself.

Akamai client (solver) source code is here: github.com/proofofbots/web-re-toolkit/tree/main/clients/akamai

Akamai client docs: https://proofofbots.github.io/web-re-toolkit/guides/akamai/

It does not use host's fingerprint, rather you provide the fingerprint profile you want akamai to use. By default it uses a Mac’s fingerprint.

This was a weekend project put together largerly with AI. There are many things I’d like to add and improve, but I do not have the time to work on this project much more than this. The docs, packages, etc are an absolute mess, I'd love if someone could contribute in cleaning them up!

I’ll be releasing Kasada’s solver too later today.

I’ve not had the chance to to throughly test it on more than a few sites, I am hoping the community (you) will help me with that. Whatever issues you run into, please make a issue in GitHub or send a message in the Discord and provide the logs + code.

Discord: https://discord.gg/nbBePnsa9

Details about how it works under the hood are available on Github: https://github.com/Joe12387/ShaderGhost

I am building a crawler and have hit a bit of a brick wall. Extracting data from tables, lists, documents and nested HTML structures is proving hard. Like really hard, especially when there is any deviation in how the information is displayed (graphs, tables, images).

I am looking for some advice or an open source crawler that I can take inspiration from.

My main issues are:

Smarter discovery - currently i find that it is very much ignoring lastmod so its not finding the most uptodate information. Im also finding that lots of information is hidden in JSON, JavaScript, structured data and they are litteraly being missed as a result

- Handling of difficult/alternative documents - PDF and DOCX with lots of complex columns confuse upon reconstruction. This is a huge one because when I manually look there are hundreds of pieces of information that are lost due to this reconstruction issue

Any advice is greatly appreciated.

Couldn't find any projects or git repos that focus on this.. anyone have any strategies for scraping public zoom links? afaik the unique code in each link is randomly generated so it would be pretty hard to brute-force guess zoom meetings.. curious to hear if anyone has tried this

I’m scraping a betting site and considering IP rotation to deal with their CloudFront 403s. I’ve tried different request rates, jitter, headers/TLS, etc., and I never go above ~20 requests/minute. I’ve also tried figuring out their rate/volume limits or any consistent pattern behind the blocks, but I haven’t found anything clearly defined. For now, rotating IPs is what has worked best for me — distributing the requests across different IPs lets me scrape for much longer, and I’m not currently running into blocking issues.

I’m not doing any login/account stuff or accessing anything private; I’m only collecting publicly available betting odds/data.

I know IP rotation is extremely common in scraping, but I’m not sure if it could be problematic in this specific context. Could rotating IPs to avoid CloudFront restrictions get me into legal trouble, or is it generally fine? I’m aware the site may not like it, but I’m mainly wondering whether there’s a legal issue with doing it this way.

Hi guys, please bear with me as I'm a complete amateur.

Is it possible for me to have a setup where I get notified when changes are made to a website including what and where the changes are?

Thanks a lot :))

I have zero experience with scraping, proxies or programming in general, so I genuinely don't know what the right approach is. I'm not asking anyone to build this for me, I just want to understand what a realistic path looks like.

My goal is that there are a handful of online auction portals selling used industrial machinery. I'd like something that goes through them every day and checks all the listings, and then notifies me only when it finds a machine tool matching a few broad criteria. Things like any CNC lathe under 10 years old, or any 5-axis machining center under 10 years old. So it's not a daily digest I'm after, the scraping runs daily, but I only want to hear about it when there's an actual match, and I don't want to be told twice about a listing I've already seen.

The listings are often messy, the year of manufacture is usually buried in a free-text description, model naming is wildly inconsistent, and the site categories are unreliable, so anything based on exact keywords isn't enough.

Two constraints are that I can't keep a machine running 24/7, and I want it based on reading the sites directly, not on the portals' own alerts or filters, which are too poor for what I need. I already pay for a Claude subscription, in case that's relevant.

How would you go about this? What should I be learning, what would you use, and any guides worth the time? Happy to share the exact portals if that makes the answers more concrete. And if there's an obvious "don't do it this way", I'd rather hear it now.

Thanks.

From my point of view data itself is not valuable on it's own, but after cleaning / organizing it / presenting it to match certain needs, it can become valuable.

But how do you find your "niche" first that you'd monetize after all the steps above?

I want to fetch the Google Maps reviews for my business and publish them on my website. After doing a bit of research, I found out that I can fetch Google Maps reviews using the Google API. However, despite two weeks having passed, I have not received a reply from Google Support. Is it possible to scrape Google Maps reviews without triggering bot detection or causing any other issues? Are there any solutions available for this?

So redditors, I’m currently working on a workflow for moving relevant data from my search results into an internal database, and the biggest headache right now is getting the information into a clean, structured format....

I’ve tested a few cloud-based extraction tools, but some seem to generate security checks or restrictions pretty quickly. I’m trying to avoid anything that sends a large volume of automated requests or interacts with the platform more than necessary....

I’m considering a local, client-side approach that only works with information already visible in the browser...

For those who’ve built similar workflows, what approach has worked best for you? Custom scripts, browser-based tools, or something simpler?

For years I could do a normal HTTP fetch. That stopped working last year because they started requiring javascript. So I upgraded to using node. That stopped working this year; I get a 403 error. Do you have any tips? Someone posted a python script here 6 years ago, so someone showed interest; of course it's out of date.

Does anyone know how soon after someone lists an item does the api pickup new listings?

I haven't used the ebay API yet but I tried using the front end and it was somewhere around 3-4 minutes. Someone competing with me for the same kind of items is somehow buying them 1-2 minutes before me. Literally a minute after the official listing time.

Any suggestions what's going on? Is that person just using the same API?

I've been building a small project to collect text from documentation for a personal AI tool. Some of them load content dynamically with JavaScript. Anyone have recommendations for a lightweight solution? I'm not trying to scrape anything huge, just a few hundred pages per sites.

i wa trying to scrape some website and get their websocket but its encrypted and ai didnt let me do it (cause of permissions) how can i make ai to be able to scrape

I’m completely new too this. Been using Codex to try to write python codes and browser automated techniques to scrape vins from Autotrader. It always gets shut down by their bot detection. The only way I’ve got results is codex doing it internally on its own browser, but because of Autotrader limiting batch sizes, it takes forever to scrape even a decent amount of vins. Any help would be appreciated.