r/WebScrapingInsider

Hi r/webscraping,

I've been working on klura — a free toolkit that gives a coding agent (Claude Code, Cursor, Claude Desktop, any MCP host) the ability to reverse-engineer a website. The agent drives a browser to complete a task once. Klura captures everything the page does underneath, then does what I call LIFT (Learn Interface From Traffic — the analysis pass) and extracts the real underlying requests and saves them as a readable, LLM-annotated JSON config, so that it can be run later - without driving the UI. The JSON config can be analyzed to understand how the site works, and can also be copied and run on other devices.

There's prior art in this space — capture-and-replay isn't a new idea. The main places klura tries to push it forward:

• page-script tier — here the saved strategy is a snippet of JavaScript that the AI codes for you. klura injects it into the live, already-authenticated page and runs it there. The pure-HTTP approach (the "convert to a requests / curl script" pattern) hits a wall on sites that bind the request to in-page state — rotating per-session CSRF tokens (GitHub's X-Fetch-Nonce is one public example), request bodies built by a JS function in the bundle, binary WebSocket transports. Reproducing those from outside the browser means porting the site's own signing/encoding logic — fragile, and often impossible. Page-script sidesteps it: the saved JS calls the page's own functions — the request signer, the encoder, the socket the page already has open — so the site does the hard part and klura just collects the result. And it isn't slow the way browser automation usually is — klura keeps a warm pool of already-open, already-authenticated pages, (configurable of course) so you don't pay browser startup per call; a page-script run is dominated by the actual request and lands close to plain-curl latency.

• A real RE + debugging toolkit the agent drives — most tools capture traffic and pattern-match. Klura hands the agent an actual JS debugger: breakpoints, stepping, live-stack inspection, WebSocket frame tooling etc. It reverse-engineers a site the way a human would — break on the function that builds the request, read the encoder, verify it reproduces. You never touch any of it; you ask in plain language. Call it vibe-RE. (Detail in How the discovery works below.)

• Self-healing — when a site changes shape and a saved strategy breaks, klura re-discovers the delta and patches the strategy automatically, instead of silently returning garbage or making you re-record from scratch.

• Runs anywhere, hands off to a human when it must — klura is an MCP server, so the same saved skill is callable from Claude Code, Claude Desktop, Cursor, Windsurf, the CLI, or a programmatic import. Skills live globally at ~/.klura/skills/<platform>/, not per-project. When a site genuinely needs a human (login, 2FA, captcha), it escalates to a remote viewer — you do that one step, the agent continues, and the saved strategy runs in the resulting authenticated session.

• Plugability — The browser driver is swappable: default is Playwright, klura ships a stealth driver that fixes the standard automation-fingerprint leaks (navigator.webdriver, canvas/WebGL consistency), and you can drop in your own. The interruption layer is pluggable too — when a site throws something up mid-flow (login, captcha), the handler that decides what happens is yours to define.

The artifact

Every skill is a single JSON file on disk. A fetch-tier example for a public search API:

{
  "strategy": "fetch",
  "method": "GET",
  "baseUrl": "https://hn.algolia.com",
  "endpoint": "/api/v1/search",
  "params": {
    "query":       "{{query}}",
    "tags":        "story",
    "hitsPerPage": "{{count}}"
  },
  "response": { "format": "json", "extract": "hits[]" }
}

The LLM that produced it leaves notes fields explaining the why of each parameter, the prereq chain, and what shape the response has. Read the file → you've reverse-engineered the site.

Execution path

Once saved, run it via klura (recommended):

klura execute hackernews search_stories --args '{"query":"show hn","count":3}'

Or you could ask your LLM again, it will prefer to execute already created strategies above creating new ones.

For page-script strategies the saved JS has to run inside the live authenticated page, so that path needs the klura runtime regardless of how you wire it up. For fetch strategies the saved JSON file gives you method + URL + body/header templates — slot in your params/cookies and curl away.

Three execution tiers

Tier	What runs	When it lands	Curl-able?
fetch	Static HTTP from Node, templated body/headers + prereq chain	Plain APIs (HN search, Amazon /s?k=, GitHub GETs)	yes
page-script	JS dispatched inside the live authenticated browser tab	Sites that bind requests to in-page state	no — page state needed
recorded-path	UI action replay through the driver	Fallback when neither above is reconstructible	no

How the discovery works

There's no recorder UI implemented or planned at the moment. The agent itself drives the browser. You tell your MCP host in plain language — "search HN for the top posts this week using klura" — and that's it. No script to write, no breakpoints to set, no JS to read. You could call it vibe-RE: you describe what you want, klura performs it and figures out how the site does it. You can also ask klura to map out a site, if you just want it to observe the API without performing a specific action.

For readers who want the how — on hard sites, the agent has access to a full RE toolkit (none of which you ever touch as a user):

• JS debugger: set_breakpoint, step, resume_execution, wait_for_pause — break on the function the page calls before dispatch, read the live stack, see the encoded payload the page is about to send.
• JS source navigation: search_js_source, read_js_function, list_loaded_scripts — find the function in the bundle that builds the body or signs the request.
• WebSocket frame tooling: inspect_ws_frame, find_in_ws_frame, explain_ws_frame_structure, get_send_encoder, try_generator_in_page — for binary protocols, capture frames live and verify the captured encoder reproduces the wire format before saving.
• Network log (get_network_log) for HTTP and WS in one feed, filterable.
• Live page inspection: a11y tree, screenshots, page text, arbitrary js_eval.

This is how klura one-shots genuinely hard flows — for instance a mainstream chat app whose message-send rides a binary MQTT-style WebSocket, with an in-page codec, snowflake IDs beyond Number.MAX_SAFE_INTEGER, and a per-session token that rotates. The agent debugs the way a human would: set a breakpoint at the call site, read the actual encoder, verify it reproduces the wire output, save it as a page-script. The same toolkit generalizes to anything that builds a signed or encoded payload in-page — rotating CSRF on graphql endpoints, signed URL params, custom binary framings.

Once on disk, the next call hits the saved skill directly — or you cat the JSON and write your own scraper.

Cost: LLM once, then never

Klura runs the LLM during discovery, then never again. After LIFT the saved skill is a plain request (or a page-script dispatch): no model, no agent loop, no tokens. The only time the LLM will need to be looped back is when self-healing is required.

A couple of concrete runs, same task, measured against a plain browser agent on the same model: a Hacker News search that costs the browser agent ~20s and ~$0.09 every time runs from the saved skill in ~270ms at zero token cost. A mainstream chat app's message-send took ~27 minutes on the first run — that's the agent reverse-engineering the binary WebSocket — but every send after is a sub-millisecond dispatch into the authenticated page. The one-time discovery cost scales with site difficulty; the warm cost is always ~zero.

Install

# As an MCP server (Claude Code)
claude mcp add klura -- npx -y @klura/mcp

As an MCP server it supports many other harnesses, please check the documentation for yours to figure out how to install it.

Repo: klura · npm: @klura/runtime, @klura/mcp

Currently at 0.2.2. Real-world feedback welcome — especially the shape of the failures.

I built a real C++ browser and gave you a TypeScript library to control it — here's why it changes scraping

Most tools like Puppeteer and Playwright bolt automation onto Chrome from the outside. They're always playing catch-up with anti-bot systems.

I took a different approach. I built the actual browser — Qt6 + Chromium engine, written in C++. Then I wrote a TypeScript library (Piggy) that controls it over a local socket. That's why Cloudflare bypasses are almost trivial and the code stays dead simple.

Two repos, one ecosystem:

🖥️ Nothing Browser (the C++ browser) https://github.com/BunElysiaReact/nothing-browser

📦 Piggy (the TS library) — https://github.com/ernest-tech-house-co-operation/nothing-browser

What you get out of the box:

🪪 Persistent TLS fingerprint identical to real Chrome — sites can't profile you

🧠 Human Mode — randomized delays, natural scrolling, no robotic timing

⚡ Socket-based IPC — millisecond latency between your script and the browser

🌐 Remote deployment — binary runs on a VPS, you scrape from local

💾 Session persistence — save/restore cookies and storage, stay logged in

🏊 Tab pooling — concurrent requests inside one browser instance

🚀 Built-in API server — one line turns your scraper into a REST endpoint with OpenAPI docs

🔄 Proxy rotation — built-in fetch, test, switch, rotate

The code looks like this:

Ts import piggy from "nothing-browser";

await piggy.launch(); await piggy.register("books", "https://books.toscrape.com"); await piggy.books.navigate();

const books = await piggy.books.evaluate(() => Array.from(document.querySelectorAll(".product_pod")).map(el => ({ title: el.querySelector("h3 a")?.getAttribute("title") ?? "", price: el.querySelector(".price_color")?.textContent?.trim() ?? "", })) );

console.log(books); await piggy.close();

That's a real browser. Not a wrapper around someone else's.

Bun-first but Node compatible. Headless and headful ship as separate binaries so you're not carrying GPU overhead when you don't need it.

📚 Docs: https://nothing-browser-docs.pages.dev

Would love issues, feedback, and ⭐ stars — built in Kenya 🇰🇪

If you have been looking for a no-browser alternative, feel free to give this a go!

Fast and lightweight.

Would love feedback or bug reports if you run it against anything weird.

I posted here before about webclaw, the Rust web extraction tool I’m building for AI agents and LLM workflows.

Since then I’ve been working mostly on the part that is least fun but matters the most in production: making extraction survive real websites.

The original version was mostly about turning pages into clean markdown/JSON quickly. That worked well on normal pages, docs, blogs, changelogs, etc.

But once people started trying it on harder targets, the pattern was obvious:

• simple fetch works until it suddenly doesn’t
• datacenter-looking traffic gets blocked fast
• some pages return a 200 with a challenge page
• some pages need JS rendering
• some sites work once and fail at scale
• some failures look like empty content instead of actual errors

So the newer architecture separates a few concerns:

1. First try the lightweight path
2. Most pages do not need a full browser. A lot of useful content is already in SSR HTML, JSON-LD, hydration payloads, or embedded data islands.
3. Detect bad responses before extraction
4. A 200 response is not always success. If the page is a bot challenge, login wall, empty shell, or blocked response, treating it as “content” is worse than failing loudly.
5. Escalate only when needed
6. Instead of using a browser for everything, webclaw tries to keep the common path cheap and fast, then escalates for pages that actually need rendering or stronger handling.
7. Return useful output, not just HTML
8. The goal is markdown, text, JSON, structured extraction, metadata, links, screenshots when needed, and clear warnings/errors.
9. Make it usable from agents
10. It ships as a CLI, REST API, SDKs, and MCP server, because Claude/Cursor/custom agents need tools they can call directly.

I’m deliberately not going to share the exact anti-bot mechanics. That is a weird arms race and posting the details publicly just burns everyone’s work faster.

But at the product level, the goal is simple:

normal pages should be fast and cheap
hard pages should not silently poison the output
agents should get clean context instead of raw page chaos

Current state:

• Rust
• AGPL-3.0
• 1,081 GitHub stars
• 124 forks
• latest release v0.5.8
• CLI, MCP server, REST API
• scrape, crawl, map, batch, extract, summarize, diff, brand, search, research
• hosted API for the parts that are painful to self-host

Curious from people here who run scrapers in production:

What do you care about more when a target blocks you?

• automatic fallback
• clear failure reason
• ability to bring your own proxy/browser
• lower cost per successful page
• raw HTML access for debugging
• screenshots
• session/sticky behavior
• something else?

Repo: https://github.com/0xMassi/webclaw
Site/docs: https://webclaw.io