u/Guiltyspark0801

Basically the title, you can mention everything that comes to your mind, whole infrastructure pricing would be nice to hear

Hey all, did you guys notice how most people jump straight into spinning up a headless browser the moment they want to grab some data off a website? It feels like the obvious move, BUT nine times out of ten you're carrying a sledgehammer where it's absolute not needed. From what I gathered, is that modern websites are essentially just JavaScript talking to APIs behind the scenes. The browser renders it pretty for humans, but the actual data? Everything is going on plain JSON through endpoints that you can hit directly. Open DevTools, poke around the Network tab for a minute, and you'd be surprised how often you find a neat little API call doing all the heavy lifting, and guess what, no browser required. For example, a plain requests.get() to the right URL will get you richer data, faster, with way less chance of getting blocked. The rendered page is often less complete than what the API returns anyway. Obviously there are cases where you genuinely need a browser, heavy client-side rendering, auth flows that are tricky to replicate, that kind of thing. But that should be the fallback and not the other way around. Have any of you made this switch or are you're still defaulting to Playwright/Selenium for everything. Do you have any specific reasons for doing so?

I've been comparing how tutorials teach APIs versus what you actually see in DevTools on sites like Amazon. The gap is embarrassing, and I don't think it gets talked about enough. What took me a while to realize is that techniques like protobuf serialization, HMAC request signing, and TLS fingerprinting aren't primarily anti-scraping measures. They exist because binary payloads are 5–10× smaller than JSON, signed requests prevent cache poisoning, and fingerprinting is cheaper than running a full bot-detection pipeline on every hit. The hostility to reverse engineering is almost a side effect. That reframe changed how I approach this entirely. Any opinions on this?