Most profitable web scraping niches today?
I used to build scrapers years ago (started back when raw HTML parsing with regex was common), but I stepped away for about 6 years.
With LLMs and agentic workflows changing how edge cases and dynamic pages are handled, I recently jumped back in. Currently, I've been running tests on entertainment data (movies, gaming, books), but I'm curious about the current commercial landscape.
For those scraping in production:
What data categories or niches are businesses actually paying for these days (e.g., dynamic e-commerce pricing, real estate, lead enrichment, finance)?
In which areas are traditional extraction pipelines struggling the most where agent-based approaches actually make sense?
Just looking to get a realistic read on how market demand has evolved.