
Scraping Reddit like it's 2025
So as of lately (as you probably experienced already), reddit shut down API access for 99% of us who don't meet their requirements and can't register an app on their platform. They also made it difficult to programmatically access their old unauthenticated json endpoints.
Now I've been running a simple workflow for a long time, and all it did was fetch latest (<24h) posts from my 5-6 favorite subs once a day in the morning, get the most upvoted ones, feed them into Google Gemini, let it create a nice little TLDR / Top 10-style recap for the most interesting/groundbreaking ones, and mail it over to me so I could read it while sipping my morning coffee.
Now that my workflow didn't work anymore, I was a bit annoyed and my coffee wasn't as pleasant anymore. So today I managed to restore it, and I'm here to (briefly) share the steps, in case anyone wants to do the same.
First of all, the problem is they block you from accessing the json endpoints (if you're unauthenticated), unless you supply appropriate headers, user-agent and cookies. The HTTP request node by itself can't do it properly, for that you need to "fake" a first visit to Reddit, get the necessary tokens/cookies populated, and only then you can access the old unauthenticated json endpoints. So I used a mix of Browserless and Puppeteer API directly on the n8n host to work around this requirement.
>Note: this only applies to self-hosted instances - In my case I run it with Docker, but the same would work even without it.
Step 1:
edit your compose.yml file to add the browserless service:
services:
n8n:
image: docker.n8n.io/n8nio/n8n
# (your regular n8n config)
browserless:
image: browserless/chrome:latest
ports:
- "3000:3000"
environment:
- MAX_CONCURRENT_SESSIONS=5
and restart your n8n stack.
Step 2:
Add the following workflow to your n8n instance: RedditScraper.json
Step 3:
In your own workflows, invoke the RedditScraper subworkflow by adding an Execute Sub-Workflow node. Set it up so that it passes a single parameter called targetURL. This will be the full Reddit URL that you want to scrape (e.g. https://www.reddit.com/r/n8n/), plain and simple.
Output will be in JSON. If you used n8n Reddit nodes previously, the output is usable in the same way - everything is in the `data` object.
If you have a faster/better way to do this, I'm all ears!