r/thewebscrapingclub

I originally built it for myself because I wanted something extremely lightweight that runs in the background like it never existed. It's called SpyWeb.

It's designed to be "set and forget." I've had it running for months on my PC tracking job boards without a single crash or memory leak.

Specific features:

• Zero Runtime: Self-contained ~7MB binary. No Python, Node, or Docker needed.
• Low Footprint: Uses <5MB RAM at idle.
• Lua Scripting: Use Lua to handle complex logic like custom headers, JS rendering, advanced monitoring, etc.
• Hot Reloading: Change a config or Lua script and the job respawns instantly, no restarts.
• Web Dashboard: Simple local UI to monitor scrape data in real-time.
• Desktop Alerts: Built-in support for system notifications and webhooks.
• Embedded DB: Built-in KV store so you don't need a separate database.
• CDP Support: Controls any Chromium or CDP-compatible browser via Lua for JS-heavy sites.
• Dual Mode: CLI for servers and a System Tray version for silent background runs.
• Deduplication: Internal database ensures you never see the same result twice.

I just released the beta with CDP integration. If you need something that just sits in the background and sips resources while actually being maintainable, check it out.

Set up is very easy and straightforward: for server-side rendered pages, it's just a few lines of config (URL, selectors, fields). For JS-heavy sites, you can write a little Lua to launch a browser and drive the workflow.

You can check it out here: https://github.com/spyweb-app/spyweb

Hey guys,
I’ve been working on browser automation that can actually survive modern anti-bot systems (especially for AI agents).
So I created a fork of Playwright for Firefox patched directly at the C++ level. It generates a different but internally consistent fingerprint per session:
• CreepJS → 0% fake
• reCAPTCHA v3 → Score 0.90
• hCaptcha → Pass
• Fingerprint Pro → bot=false, tampering=false

Repo: https://github.com/feder-cr/invisible_playwright

If you’re fighting heavy anti-bot protection or building resilient agents, I’d love to hear your thoughts or test results. Feedback, issues, and contributions are very welcome!

Thanks in advance 🚀

Runo is a web-scraping API that returns typed, structured JSON. You define a schema (field name, type, example value), and Runo fetches the page and returns the data. No HTML, no parsers, no post-processing.

Over the past few weeks, I have been building this non stop. Currently, every scraper API out there solves the site fetching problem but left the extraction of the actual data entirely to users. Runo makes that completely disappear.

For Runo, I went ahead and added JS rendering, stealth mode, and full LLM extraction to make this a fully functional and capable of scraping most if not all sites.

Also, another major problem with current web scrapers is that they charge per feature or bundle them into expensive credit tiers. A single large or JS rendered request can cost 5-75 credits, which means you essentially get nothing out of their plans. Runo is flat per request, no matter the site. At the Scale tier, Runo works out to $0.90 per 1,000 effective requests vs. around $6 for the nearest Firecrawl equivalent. My jaw dropped when I was testing Runo and came across these numbers.

You can check it out here. I created a free tier that is 500 requests/month, no credit card required. Take it for a spin and let me what can be improved. I would love feedback.

I built a real C++ browser and gave you a TypeScript library to control it — here's why it changes scraping

Most tools like Puppeteer and Playwright bolt automation onto Chrome from the outside. They're always playing catch-up with anti-bot systems.

I took a different approach. I built the actual browser — Qt6 + Chromium engine, written in C++. Then I wrote a TypeScript library (Piggy) that controls it over a local socket. That's why Cloudflare bypasses are almost trivial and the code stays dead simple.

Two repos, one ecosystem:

🖥️ Nothing Browser (the C++ browser) https://github.com/BunElysiaReact/nothing-browser

📦 Piggy (the TS library) — https://github.com/ernest-tech-house-co-operation/nothing-browser

What you get out of the box:

🪪 Persistent TLS fingerprint identical to real Chrome — sites can't profile you

🧠 Human Mode — randomized delays, natural scrolling, no robotic timing

⚡ Socket-based IPC — millisecond latency between your script and the browser

🌐 Remote deployment — binary runs on a VPS, you scrape from local

💾 Session persistence — save/restore cookies and storage, stay logged in

🏊 Tab pooling — concurrent requests inside one browser instance

🚀 Built-in API server — one line turns your scraper into a REST endpoint with OpenAPI docs

🔄 Proxy rotation — built-in fetch, test, switch, rotate

The code looks like this:

Ts import piggy from "nothing-browser";

await piggy.launch(); await piggy.register("books", "https://books.toscrape.com"); await piggy.books.navigate();

const books = await piggy.books.evaluate(() => Array.from(document.querySelectorAll(".product_pod")).map(el => ({ title: el.querySelector("h3 a")?.getAttribute("title") ?? "", price: el.querySelector(".price_color")?.textContent?.trim() ?? "", })) );

console.log(books); await piggy.close();

That's a real browser. Not a wrapper around someone else's.

Bun-first but Node compatible. Headless and headful ship as separate binaries so you're not carrying GPU overhead when you don't need it.

📚 Docs: https://nothing-browser-docs.pages.dev

Would love issues, feedback, and ⭐ stars — built in Kenya 🇰🇪

If you have been looking for a no-browser alternative, feel free to give this a go!

Fast and lightweight.

Would love feedback or bug reports if you run it against anything weird.