Is Web Scraping Legal? An Internet Law Professor’s Insights on Public Data, CFAA, Copyright and Terms of Service | AMA with Eric Goldman
Hey everyone,
I'm Ian Kerins, CEO & Co-Founder of ScrapeOps.io.
We're back with AMA #12, and this time we're taking on the question that comes up in nearly every scraping thread: is web scraping actually legal?
This Friday, October 9, at 11:00 AM PT (9:00 PM GMT+3), we'll be joined by Eric Goldman, Associate Dean for Research and Professor at Santa Clara University School of Law, and Co-Director of the Datta Center for High Tech Law, for a live AMA on r/WebScrapingInsider.
Eric also co-supervises Santa Clara's Privacy Law Certificate. You can check out his work on his website, blog and Bluesky.
A few years ago, the scraping law conversation was mostly about hiQ v. LinkedIn and whether accessing public data violates the CFAA. Since then, courts have been dealing with contracts, copyright preemption, DMCA anti-circumvention and other legal theories, and the recent SerpApi cases raise a new question: can information be public to humans while automated access to it is still legally restricted?
>So is the law getting clearer, or has the uncertainty just moved somewhere else?
Eric's focus is US law, so expect the discussion to be primarily US-focused. We'll balance the bigger picture of where scraping law is heading with the practical questions facing companies that scrape data, provide scraping infrastructure or build AI agents.
We can dig into:
- How scraping law has evolved. what's reasonably settled after hiQ, what's still open, and whether things are getting more predictable or more fragmented
- Assessing a scraping operation. the questions to work through, the facts that push a project from low-risk into dangerous, and any red lines
- What "publicly available" means now. public to a human vs. public to a bot, and whether it matters if you go direct, through Google or through an authorized user's account
- Anti-bot measures and the law. logins, CAPTCHAs, bot detection, and which techniques (rotating IPs, residential proxies, fingerprint manipulation, CAPTCHA solving) change the legal analysis
- Terms of service and cease-and-desist letters. when you're actually bound, what changes once you're blocked or sent a notice, and Reddit's browsewrap allegations in the Anthropic case
- The recent cases, compared. Google v. SerpApi, Reddit v. SerpApi, Oxylabs, AWMProxy and Perplexity, and Reddit v. Anthropic
- Platforms and data they don't own. how platforms control public user-generated content through licenses, contracts and technical measures
- How you use the data. whether the same collection method can be fine for one purpose and unlawful for another, from price monitoring to resale to AI training
- AI agents. whether software acting for a user who already has access is also authorized, with Amazon v. Perplexity as an entry point
- What to watch next. the legal developments scraping companies should track over the next 3 to 5 years, and which risks are overrated or underestimated
We can discuss cases like Van Buren, hiQ v. LinkedIn, Meta v. Bright Data, X v. Bright Data, Meta v. Voyager, Compulife v. Newman and Ryanair v. Booking.com, alongside the ones above.
If you've ever wondered "Does a ToS I never agreed to still apply to my scraper?", "Does putting data behind a CAPTCHA or login change whether scraping it is legal?", or "Is an AI agent using my customer's account legally different from the customer doing the same thing by hand?", this is a good one to ask.
We've now had eleven AMAs with the community, and the conversations have covered a pretty wide part of the scraping stack.
- Our first AMA covered proxy infrastructure, Cloudflare, browser automation and scaling scrapers.
- Our second with WebClaw explored AI agents, hidden APIs, open-source scraping and LLM infrastructure.
- Our third with CloakBrowser went deep on stealth Chromium, fingerprinting, anti-bot detection and browser automation.
- Our fourth with Browser Use brought the conversation into AI browser agents, AI-powered scraping, evaluations and browser infrastructure.
- Our fifth with Stan Sadokov from NodeMaven focused on residential proxy quality, IP reputation, sourcing, pricing and what actually makes one proxy network better than another.
- Our sixth with Saksham Solanki, creator of HTTP Cloak, went down to the protocol level, covering TLS fingerprints, JA3/JA4, HTTP/2, HTTP/3 and why scrapers can get blocked even when their proxies are fine.
- Our seventh with Huey from BrowserAct explored AI-powered web scraping, natural-language workflows, browser automation and the move toward no-code scraping.
- Our eighth with yours truly brought the discussion back to proxy infrastructure, with real-world insights from testing 50+ providers across billions of requests.
- Our ninth with Jean-Patrick Bisson from Geonode dug into whether "unlimited" proxy models can really deliver, and what buyers should look at beyond the word "unlimited."
- Our tenth with Scrape.do went straight at staying unblocked on difficult, high-defense websites at real scale.
- And our eleventh with Ibrahim El Khalil MLATA went straight to the anti-bot systems themselves, from bypassing Akamai, Cloudflare, DataDome and Imperva to mobile app reverse engineering with Frida, running scraping at scale and AI agents that write and repair their own scrapers.
Now for #12, we're moving from the technical side of scraping to the legal side: what the law actually says about scraping, and where it's heading. If you're building web scrapers, data pipelines, scraping APIs, proxy infrastructure or AI agents, come join the discussion.
As with any legal topic, this is general discussion and education, not legal advice for your specific situation.
Drop your questions below.
Eric will be answering them during the AMA.
Looking forward to another good one.
Ian