u/BlueLagoon226

I’m hitting a wall on a heavy JS site and it’s making me rethink my whole approach when starting a new scraper.

Usually, my first move is opening up the network tab, filtering by XHR, and trying to mimic the backend API requests to get clean JSON. It’s fast and doesn't melt my RAM like a browser fleet does. But lately, I’ve been running into sites with massive header obfuscation or dynamic tokens that change every few minutes. I just wasted half a day trying to replicate a single request before realizing Playwright would have finished the job in twenty minutes.

There's only so much time you can spend searching for tricks on how to find hidden api endpoints for scraping before it just makes more sense to eat the infrastructure cost of running headless browsers.

Where do you guys draw the line? Do you have a hard cutoff time where you give up on the network tab, or do you exhaust the API route no matter how long it takes?

I’ve been seeing people argue XPath vs CSS selectors like it’s some huge loyalty thing, so I’m curious how people here actually use them once a scraper gets past the quick test stage.

I used to default to CSS selectors because they felt easier to read while I was building. They still feel cleaner to me when the page has decent class names or a simple structure. The problem is that a lot of sites do not make it that easy. Sometimes the useful data is sitting near a label, inside a weird table, or buried in markup that clearly wasn’t written with scraping in mind.

That’s where XPath started making more sense to me. Not as my default for everything, but as the thing I reach for when CSS starts feeling like I’m forcing it. At this point I don’t really care which one is supposed to be better. I care about what I can come back to later without hating myself when the site changes.

I used to avoid JavaScript-heavy sites like the plague when I first started scraping.

They felt like a trap. You’d open the page source and half the stuff you wanted just wasn’t there, so I assumed the only option was firing up a browser and hoping for the best. The thing that changed it for me was learning to watch the Network tab. Once you see where the page is actually pulling the data from, a lot of these sites become way less scary.

Sometimes you still need Playwright. No way around it. But I try to find the background request first before turning the scraper into a full browser job.

The age-old question, and somehow it still starts arguments every time it comes up. I don’t think there’s a perfect answer, but there are definitely wrong tools for the job.

BeautifulSoup is still my go-to when the page is simple and I just need to pull data from HTML without turning it into a whole project. Scrapy makes more sense when the crawl starts getting serious, especially when there are more pages involved or the job needs to run without falling apart halfway through. Playwright is what I reach for when the site basically refuses to behave without a real browser. I try not to start there though, because it’s heavier than people expect.

I’ve used all 3 in the same kind of workflow before. Scrapy handles the main crawl, BeautifulSoup helps parse some annoying HTML pieces, then Playwright only steps in for pages that actually need rendering. That setup usually feels cleaner than forcing one tool to do everything.

Forget budgets or limitations for a second. What would your “all-star” scraping pipeline look like?

Mine would be a setup where the boring stuff is handled properly. Clean fetch logic, browser fallback only when needed, automatic retries that don’t make things worse, simple monitoring, and storage that doesn’t turn into a junk drawer after a week.

Not the flashiest answer, but I’d take a scraper that is easy to understand six months later over some giant overbuilt mess. You don't want to see the nightmare that was my first scraper.

AI agents can help get the ball rolling, but I don’t think they work as the final approach.

I’ve seen people treat them like they can just hand over a finished scraper on the first go. The first draft might look decent, but once you test it you still have to clean up the logic and figure out what it misunderstood.

Sometimes the back and forth takes just as long as writing it yourself. At the end of the day its still just a tool to help with some gaps but it shouldn't be blindly trusted.

I’ve been seeing a lot more Playwright lately, but still plenty of people sticking with Requests/BS4 or Scrapy when the site doesn’t need a browser.

I’m mostly using Python with Requests and BS4 for simple stuff, then Playwright when a site forces it.

Always interesting to see what people actually use once the scraper has to run more than once.