u/External-Wealth3756

I’m looking to connect with marketers focused on developer and data scraping audiences to exchange thoughts on long-term affiliate partnerships in this niche.

The proxy service I work with offers a combined upfront and recurring commission framework for partners, with all tracking and payouts managed through a third-party affiliate platform. This space has steady repeat customer demand, making it viable for sustained passive income over time.

If you’re already active within dev, automation or data collection affiliate spaces and would like to discuss relevant vertical options, feel free to share your thoughts in the comments.

I’m working on a small side project where I try to collect Amazon product reviews for a few specific items (mainly for sentiment analysis and internal research).

The idea is pretty simple:

• input a list of product URLs
• scrape recent reviews (rating, text, date)
• store them in a small database for analysis

I can get it working locally with Python + Playwright, but once I run it for more than a few products, I start running into issues like:

• requests getting blocked halfway through
• CAPTCHA pages appearing randomly
• some products working fine, others failing immediately
• inconsistent results even with the same script

I tried slowing things down and adding retries, but it still doesn’t feel stable enough for real usage.

From what I understand, this might be related to IP reputation or request patterns, so people keep mentioning proxies.

But I’m honestly not sure what I should be using for something like this.

Since it’s just a small project (not large-scale scraping), what type of proxy setup would make sense here?
Is residential really necessary, or is that overkill at this stage?

Demo scripts usually run fine, but once you move to production, many proxy setups fail because of session continuity and block handling.

Here’s what we’ve learned from scaling Playwright crawlers:

• Sticky sessions matter more than IP count: rotating proxies every request often triggers CAPTCHAs. Keeping an IP bound to a session for a sequence of requests reduces failures.
• Residential IPs for realistic reputation: Datacenter proxies may work for demos but get blocked quickly at scale. Residential proxies, like those from Novada, maintain more natural request patterns.
• Adapt rotation to target behavior: not every page needs a new IP. Adjust rotation frequency based on rate limits and observed errors.
• CAPTCHA mitigation: combine IP rotation with session persistence and slow, randomized request patterns. Even basic JavaScript-rendered pages are more stable this way.

In short: the framework (Playwright) is just one part — proxy quality + session management + CAPTCHA-aware rotation determine whether your production scraper actually keeps running.

Would love to hear how others handle long-running Playwright crawls under heavy anti-bot defenses.