Last month I spent a day and a half (i know!) debugging a scraper that wasn't broken. The logs said 200 OK all the way down; the output had a German page priced in dollars, a "student price" that was actually retail after a silent bounce, and a 99,90 € "price" that was the monthly financing installment, not the laptop. Every request "succeeded." No row was usable though.

A status code only proves the server said something. Pages redirect silently, geos fall back, currencies switch, prices render client-side into a valid empty shell, all under a cheerful 200. So I validate content, not connections. For this I reached for residential proxies for geo-testing from Proxy-Seller, partly because they were already set up, but mostly because localized pages only behave honestly when the IP looks like a real local visitor.

Picking a target

First idea: Apple. But apple.com kills a plain requestsclient at the TLS handshake: no status, no HTML, nothing to parse. microsoft.com, same story with better manners. Not the proxies' fault, the same IPs fetched everything else instantly. Big storefronts fingerprint the handshake itself, and Python's doesn't look like Chrome's, whatever the User-Agent claims. Getting past that means TLS-fingerprint impersonation tooling, and I didn't want to go deeper, cuz I'd rather be a good citizen and pick a site that doesn't mind. :D

Lenovo doesn't: public store pages served whole to a script, plus a student storefront everywhere: us/en/d/deals/student/, and d/student-laptops/shop-by-grade/ under gb/en, de/de, fr/fr (also pt/pt and friends).

The test

One product, the ThinkPad X1 Carbon, same /p/... path in every country store, through proxies in 4 countries, on the regular store and the student store, where silent redirects and almost-prices hide. Each response is checked for status and final URL, block markers, the expected currency, the right product with a machine-readable price (from Lenovo's schema.org JSON-LD buy-box offer, not a money-shaped regex match), and, on the edu channel, proof it's an education page with a usable price, not a 1,00€ placeholder or /mo. installment. Only then: valid_scrape: true.

The core of the validation, trimmed to the essential lines:

# price from schema.org JSON-LD buy-box offer, not a money-shaped regex
def ld_offer(raw):
    for m in LDJSON.finditer(raw):
        data = json.loads(m.group(1))
        for it in (data if isinstance(data, list) else [data]):
            if it.get("@type") == "Product" and (it.get("offers") or {}).get("price"):
                o = it["offers"]
                return it.get("name", ""), float(o["price"]), o.get("priceCurrency", "")
    return "", None, ""

r = s.get(url, headers=UA, timeout=60)          # s.proxies set per country
name, price, iso = ld_offer(r.text)
row = {
    "status": r.status_code,
    "final_url": r.url,                          # catch silent redirects
    "currency_ok": iso == cfg["iso"],            # right currency for the country
    "price": price or "",                        # empty when the page has none
}
row["valid_scrape"] = (row["status"] == 200 and row["currency_ok"]
                       and bool(row["price"]))

* Trimmed for readability, full runnable script here.

What came back

Live run, 12 June 2026.

So, was 200 enough?

Nope. Two flavors of failure.

The loud one. The US store returned 403 Access Denied on the product page, while the same IP fetched the US student page seconds later. The IP isn't the problem, residential Spectrum, Alabama, as legit as traffic gets; /p/ pages just run a stricter bot policy than /d/, judging client, not address. At least a 403 shows in logs.

The quiet one. All four student pages returned 200, say "student" in the right language, show the right currency, weigh up to 1.2 MB, and contain zero prices. Student pricing renders client-side; the HTML on the wire is a valid, offer-free shell.

Check only "200 + currency symbol" and you log eight successes tonight and an empty half-dataset at report time. The sole price-shaped string in the German page: 1,00€, a placeholder begging a sloppy regex to call it a student price.

The passing rows are the payoff: the same ThinkPad costs £2,560.00 in the UK, €2,345.49 in Germany, €2,528.10 in France, a €180 spread between two eurozone stores. That signal only exists if every request truly exits in the right country, and the proxies were flawless: correct geo, sub-second warm responses (0.21 s from Germany!), full pages wherever scripts were welcome. Every failure above is content or site policy, never the network.

The fix isn't clever, just specific: check the final URL, demand the expected currency, prefer JSON-LD offers over regex matches, make an edu page prove it, and let "no price in the HTML" be a result your pipeline can express. Five cheap assertions turn invisible failures into loud ones.

And also, picking the right residential proxies matters just as much as the validation itself. On that front, my proxies kept the network layer boring enough that every failure I caught was a content problem, not a block. After enough runs like this, I'd say Proxy-Seller offers some of the best residential proxies for scraping I've used. It's still my go-to for this kind of work.

So, tell me, do you validate beyond the status code? My dumbest "200 OK but useless" story is now a 1.2-megabyte student page without a single price in it. What's yours?

In my experience a lot of "residential" pools are partly datacenter IPs sold under a residential label, and sites that block automation recognize those hosting ranges no matter what the provider called them. You can check it by hand in a few minutes.

The thing I go by is the ASN, the network an IP belongs to. A real residential IP usually sits on a consumer broadband provider. A datacenter IP sits on a hosting or cloud network.

1. Set the proxy in your browser, then open any "what is my IP" page through it to get the exit IP.
2. Look that IP up on a lookup site like IPinfo and read the ASN or org line, for example "AS7922 Comcast Cable".
3. A consumer ISP name usually means residential. A hosting or cloud name points to a datacenter IP.
4. Repeat across 30 to 50 rotated exits. In my runs a real pool spreads over many ISPs, while a fake one sits on a few hosting networks.

Important note: I would not trust a single service's usage type field, since they disagree often enough that the raw ASN is the safer thing to go by.

I also wrote a Python script that does this check automatically. Let me know if you're interested, I can share it!

How do the rest of you check a pool before you trust it?

This one cost me a chunk of a dataset before I noticed, so here is the short version.

I was pulling a few hundred product pages. A clean end to the run. All requests returned 200, no errors in the log. When I opened the output, around 15% of the rows were identical, and all of them matched the site's homepage, not a product.

Here is what happened. Some of the product URLs led to items that no longer existed. Instead of a 404, the site silently redirected those requests to the homepage and responded 200 there. requests follows redirects by default, so my code never saw the hop. It got a 200 and a whole HTML page, parsed it, wrote a row. The row was actual HTML, just from the wrong page.

The status code only tells you that the last server in the chain replied. It does not say which URL actually answered. To trust a row, you have to confirm the final URL, not just that something came back.

The fix is a comparison. After the request, read response.url and check that it still contains the path you requested. If it does not, treat the row as a failure, not data.

import requests

HEADERS = {"User-Agent": "Mozilla/5.0 ... Chrome/124.0.0.0 Safari/537.36"}

def fetch(session, url, expected_path):
    r = session.get(url, headers=HEADERS, timeout=30)
    if r.status_code != 200:
        return {"url": url, "ok": False, "reason": f"status {r.status_code}"}
    if expected_path not in r.url:                 # quietly redirected elsewhere
        return {"url": url, "ok": False, "reason": f"redirected to {r.url}"}
    return {"url": url, "ok": True, "html": r.text}

session = requests.Session()
for path in product_paths:
    target = f"https://shop.example/p/{path}"
    row = fetch(session, target, expected_path=f"/p/{path}")
    print(row["url"], row["ok"], row.get("reason", ""))

Two things I learned from that. First, allow_redirects=False is not the solution in itself, because many redirects are fine and you want to follow them. The intent is to check the destination, not to stop the hop. Second, a redirect to a login page or a default catalog page looks exactly the same in your logs as a real result, so the check is worth adding the first time a site changes its routing.

Do you compare the requested URL against the final one, or do you trust the status code and the body? I want to know if there is a cleaner pattern than a substring check, because mine feels basic and there are probably edge cases I have not hit yet.

I’m trying to understand how others make this decision. How do you decide that? Is it mostly the money, the time, how often the data is updated or how likely the site is to block you?

I feel like a lot of people understand the basic security advice but still skip the parts that actually protect them. They know the rules and just don't follow them.
The one I run into most is password reuse. Same password across a dozen sites, and when one of those sites gets breached, the rest are open too.

Which habits you think people should take more seriously? And have you ever found a way to explain it that actually got someone to change what they do?

A few years ago I was brought in to basically run paid social. That was the job. Over time it became paid social + email, then + landing pages, then + reporting and now I’m also expected to know enough AI tools to "3x my output" with the same title and roughly the same salary.

I think the AI can make the work harder, not make it easier. You have to learn each tool, how to set it up properly, and that in itself takes real time. And even with everything set up and working well, the AI can suddenly start turning out worse results out of nowhere, and then you're back to editing prompts and fiddling with settings to get it back to where it was.

So instead of saving time, it often just adds a new layer of stuff to manage, on top of everything else the role has already expanded into.

Is this happening to anyone else? Or did I end up in the wrong place? And if you resisted the expanding scope, did it really make a difference?