r/scrapingtheweb

Hey everyone,

I'm building a personal price-comparison app in react native where users can scan a barcode or search for a product, and the app compares the prices across multiple Dutch supermarkets (like Albert Heijn, Jumbo, Dirk, etc.).

My backend is built with Flask and hosted for free on Vercel. However, I'm running into the classic cloud-hosting wall: almost all major supermarket websites block or throw 403 forbidden errors on standard requests from Vercel's datacenter IPs due to anti-bot protection.

Since I want to keep this lightweight and free (serverless), I'm looking for architectural advice on how people usually build multi-supermarket scrapers or price checkers:

• What are the best free or low-cost ways to bypass these blocks for multiple different domains?
• Are there alternative public endpoints, unofficial APIs, or lightweight proxy setups that make it possible to aggregate prices from multiple grocery chains?

Any architectural tips, code patterns, or alternative approaches would be super helpful!

I’m designing a system where many users can create their own independent search agents.
Each agent has its own search criteria and runs periodically to retrieve fresh results.
The challenge is that every user can have multiple agents, and each agent can perform multiple different searches. As the number of users grows, the number of individual searches grows very quickly.
I need the searches to remain independent and reasonably real-time, while also preventing the system from becoming overloaded as the number of users and agents increases.
How would you architect the scheduling and execution layer for this kind of system so that it can scale to a very large number of users and recurring searches?
I’m particularly interested in how you would approach scheduling, concurrency, backpressure, and prioritization without letting a traditional queue grow indefinitely.

I'm trying to understand when it's better to build in concurrency and adaptive pacing myself, versus when you can just use a provider that handles the platform-specific scraping/collection for you.

I started learning web scraping, two weeks in now. I am making some progress, built a little python script to pull product listings off a couple sites for practice. Works fine for like 20 requests then boom, banned or captcha city. Apparently I need to switch up my proxy. I did some reading and now I'm more lost than before. Residential, datacenter, mobile, private proxy... no clue what actually matters vs whats just marketing fluff. I am still new to this and not looking for anyone to promote their businesses on this post, I need genuine advice.

A few weeks back I set up kind of a trap on one of my sites (side projects), I hid a link to a fake git repo inside an html comment on a page that no one would probably ever read, and that repo had some bogus endpoints in a shell script. nothing linked to it anywhere else, no sitemap, nothing. Now, fast forward almost three weeks and I start seeing hits on those exact fake endpoints coming from Amazon IPs, (I checked the official amazonbot ip range list) and yep, what do you think? IT MATCHED THEM. The thing is my robots.txt explicitly disallows amazonbot, has for a while. So either they don't check it or don't care. What really got me is it wasn't using the amazonbot user agent when it hit those endpoints, just to what looked to me was generic browser string. So it's not even pretending to follow the rules, it's actively hiding what it is while going after stuff hidden in source code that a normal crawler would have no business finding. Also weirdly it didn't touch links inside the fake markdown docs I had set up there, only pulled the url out of the shell script, and it waited a full day before trying it, then tried again a day after that. Maybe it was like parsing code for training data on a delay? I am not too sure what their take was. I've checked the net for similar situations and it's not illegal to do so, but I just wanted to vent because I always assumed the big names were at least somewhat well behaved compared to the sketchy scrapers, apparently not.

TLDR: Amazonbot ignored my robots.txt and used a fake user agent to hit an endpoint that was never linked anywhere

We built a news search API for RAG — looking for feedback

We have been working on a new News Search API at Webz.io, specifically with RAG and AI agents in mind.

The basic idea is simple as you know is using natural-language query like:

>

The API searches Webz.io’s news data and returns the most relevant articles and matching content chunks.

You can also filter by date, country, language, source, sentiment, and category.

For RAG, the content chunks are probably the part I’m most interested in. You can feed the relevant section into your context rather than passing an entire article through the pipeline.

We also built an MCP server, so agents can search the news directly without having to build a separate API integration.

Docs: https://docs.webz.io/docs/webz/news-search

I’d be interested in feedback from people building RAG systems: when retrieving news, what matters more to you: retrieval quality, freshness, source coverage, or getting smaller relevant chunks back?

Right now i am mkaing a vinted tool scraping displaying and more, thats not the point but how do i learn the stuff like maby proxies, idk what type of scraping if its for multiple people from a server and the other stuff i need pls help me !

I'm curious, how much do you guys ask for a project like this in the European market?

Scraping 20k+ products from an e-commerce website, including product data and images. The products also need to be translated, with the titles and descriptions rewritten and optimized for SEO.

I'm a developer myself, I'm curious, how much would you charge for a project like this in the European market?

Thanks in advance.

Quick update for anyone still tracking this. After the FBI/IRS-CI seizure back in July and the whole Popa botnet mess, netnut.com and netnut.io are still behind the seizure banner, but Alarum just quietly stood up a new site at netnut.ai. New branding, new positioning too, it's leaning hard into the "AI era" now with an LLM scraper API and RAG/agent framing instead of the old residential proxy messaging, (they still sell those, just not being pushy about it.)

No public statement yet about the Popa allegations or what changed on the sourcing side. But still, the company took a hit and lost thousands of clients because of this mess. Will update if Alarum puts out anything official

I’m looking for something that can pull Walmart product data without turning into a constant maintenance project.

Open source would be ideal, but I’m fine with a paid tool if it genuinely works. I’m mainly interested in product titles, prices, availability, seller info, reviews, and product URLs.

There are plenty of “best Walmart scraper” lists out there, but most seem to repeat the same tools without saying whether anyone actually ran them against Walmart recently.

If you’ve personally used one in 2026, what did you try?

It would be useful to know roughly what you tested and what started failing first. Blocks, missing fields, location-specific prices, cost, or something else.

Not looking for affiliate links or tools people have only read about. I’d rather hear about an imperfect tool someone actually tested.

​

Salut! Caut pe cineva care să mă ajute cu un proiect relativ simplu de web scraping/automatizare.

Am o listă de aproximativ 50–100 de nume și vreau un tool care să le caute periodic pe un site de licitații și să-mi arate când apar rezultate noi, împreună cu informațiile de bază (preț/estimare, imagine, data licitației și link).

Lista de nume trebuie să poată fi modificată ușor.

Pentru început vreau să facem un test pe un singur site. Dacă funcționează bine, aș vrea ulterior să adăugăm și alte site-uri.

Nu caut o aplicație complicată sau ceva foarte scump, ci o soluție simplă și funcțională.

Dacă ați mai făcut proiecte de scraping/automatizare în Python, dați-mi un mesaj în privat cu un cost aproximativ și, dacă aveți, un exemplu de proiect similar.

Mulțumesc!

I’ve been using Hermes for quite a while now, and so far it hasn’t felt particularly useful for me. I want to start using it more seriously and actually automate parts of my work with it.
I’m looking for any good skills/plugins I can add that would let Hermes:
• Generate leads for me
• Write and automatically send emails
• Edit videos
Right now it can’t edit videos, can’t write + auto-send emails, and can’t generate leads.
I also need solid frontend + backend skills (or n8n community nodes / similar) so it can build websites for me with a proper, clean frontend. Currently the frontend it produces isn’t good enough.
Two more practical questions:

1. What’s the best browser I should give Hermes access to so it can browse the internet smoothly and see everything properly?
2. I’ve already given it full access to my computer. When it tries to use my computer (like Cursor does — websites, browser, etc.), the flow is not smooth. It keeps hitting problems and errors. What can I do so it can use everything on my machine cleanly and without constant issues, since it already has complete access?
   Any recommended skills, setups, browser suggestions, or workflow tips would be really helpful.

The tricky thing with screen scraping is that most people reach for it when they don't need it.

The difference

Web scraping reads a page's HTML or an API. The data arrives already structured, tags and values you can grab directly. Screen scraping reads what's rendered on screen, so the data arrives as pixels and you need OCR to turn it back into text.

https://preview.redd.it/ovo6pbl07zih1.png?width=903&format=png&auto=webp&s=9fe453509e0405de1b4051afc1392b7d5a4458fe

Why the extra step matters

More stages means more places for things to break, and OCR errors are sneaky. In a quick test with Tesseract, clean text read at 100% accuracy, but a slightly low-res capture hit 96% and still flipped a price from 1299.00 to 2299.00. A blurry one turned a 4.6 rating into 46. The headline accuracy looks fine while single digits quietly corrupt your dataset.

When to actually use it

If the data exists in the HTML or an API, parse that. Screen scraping is for when it genuinely only exists visually, like prices rendered as images, embedded charts, scanned PDFs, or legacy terminal systems with no API at all.

If you do go the OCR route

Capture tight regions instead of full pages. If the number you need sits in one panel, screenshot that panel. Less noise means cleaner OCR output, and small layout changes elsewhere won't break your job.

https://preview.redd.it/m7zo8tj17zih1.png?width=970&format=png&auto=webp&s=08c196ae71b80971a0fc1368691f3b9ec11c6337

Validate output against expected formats, especially numbers. Don't trust raw OCR text for anything that ends up in a dataset.

For JS-heavy pages, render in a headless browser first, then capture. A plain HTTP request to a single-page app returns an empty shell.

One last thing

Budget for maintenance. A code-level parser breaks when the markup changes, a screen scraper breaks when the layout changes, and layouts change more often.

Curious if anyone here has run OCR pipelines at scale, and how you handle validation for numeric fields.

https://preview.redd.it/s9iys8jgpjih1.png?width=2326&format=png&auto=webp&s=19615df29a5277b3500da78621fa67a7ed651570

I check Google Shopping for a handful of products fairly often and never write anything down, so a month later I have no record of whether a price actually moved or whether I imagined it.

This writes it down. Every Monday it runs one Google Shopping search per product on your list and appends a single summary row per product: lowest price, median price, highest price, the cheapest seller by name, and the biggest discount showing that week.

Schedule → split the product list → Apify Google Shopping → drop products with no offers → aggregate → Google Sheets

Median is the column I actually use. The lowest price on a Google Shopping page is frequently a listing for the wrong variant, a refurb, or a seller nobody would buy from, so a chart of lowest price is mostly noise. The median moves when the market moves. The workflow trims the bottom outliers before it aggregates for exactly that reason.

A few ways people use this shape of data.

• If you sell the product, you get a weekly record of where your price sits against everyone else ranking on the same query.
• If you are buying something expensive and not in a hurry, you get the price history Google Shopping does not keep for you.
• If you source or resell, the cheapest-seller column is the interesting one. Who keeps winning on price week after week tells you more than the price does.

No AI model anywhere in it, so there is nothing to pay OpenAI for and no number that can be hallucinated. Cost is the Apify side only, roughly four cents per product per week at one page of about 40 listings each.

Template: Track weekly Google Shopping prices and log summaries to Google Sheets

It calls a Google Shopping API I wrote.

Limits worth knowing before you set it up:

• Google Shopping results are localized, so the same query from a different country returns different sellers at different prices; the template pins country and language so your history stays comparable week to week.
• It reads the search page and not individual merchant pages, so there is no stock level and no per-seller shipping cost.
• And it is a weekly snapshot, so a two-day flash sale that starts on a Wednesday is invisible to it.

Two things I keep going back and forth on adding: a week-over-week change column so the sheet flags movement instead of you scanning for it, and an alert when a named competitor undercuts your own price.

Would either of those be useful, or is there a field you would want in the row that is not in there?

Since May and the old reddit almost gone, what are currently the best libs on reddit?
What is your favorite?
I want to scrape all the posts of my top favorite 100 (small) subreddits and sort them by AI to delete AI slop + self promotion and really only get the interesting posts for me to read.

What do you suggest?

I'm doing a research project, need to pull maybe 5-10k google search results like queries, some news results, a bit of maps data mixed in.
Tried rolling my own with requests + bs4 first weekend, captchas within like 20 requests. Switched to headless chrome, still got blocked eventually. This is a one off pull, not a long running service. I don't want to set up rotating proxies and retry logic for something I run once and throw away.
What's the best serp api right now? Something that handles the google search side, news too ideally, without me babysitting captchas constantly. Budget is small. This isn't a company thing, just me lol

I run freehire, an open-source job aggregator that ingests postings from dozens of ATS platforms (Greenhouse, Lever, Ashby, Workday, etc.), normalizes them into one schema, and runs each through a facet pipeline.

Just exported the whole catalogue to Hugging Face: freehire-jobs — 6,041,471 postings.

Each row is a raw posting (title, company, description, URL, source, posted_at) plus derived facets:

- Dictionary-only (deterministic, no LLM): skills, seniority, category, work_mode, posting_language, employment_type, education_level, english_level, experience_years_min

- Dictionary-first, LLM-filled: countries, regions, cities

- LLM-only: salary_min/max, currency, period — pay is stated in every format imaginable, no dictionary handles that

Company rows carry their own facets too (industries, HQ country, size, YC batch/stage where applicable).

Format: 20 gzip JSONL shards, split by row range for easy streaming/resuming.

Backstory: I originally wanted to train a cheap classifier for seniority/category tagging instead of paying for an LLM call per job. Turned out the dictionary+LLM pipeline already in prod beats a from-scratch classifier by enough that finishing the classifier wasn't worth it — so I'm sharing the data instead.

Hello everyone I am working on an utility tool (tcg scanner like collectr and price charting) and I need to fetch daily prices of 8000 cards on eBay with with different variations such as Raw , PSA 8 9 10 .. which makes it 40000 request , did anyone work on something like this or is there any services that gets data from eBay with this amount ?