Hi r/webscraping,

I've been working on klura — a free toolkit that gives a coding agent (Claude Code, Cursor, Claude Desktop, any MCP host) the ability to reverse-engineer a website. The agent drives a browser to complete a task once. Klura captures everything the page does underneath, then does what I call LIFT (Learn Interface From Traffic — the analysis pass) and extracts the real underlying requests and saves them as a readable, LLM-annotated JSON config, so that it can be run later - without driving the UI. The JSON config can be analyzed to understand how the site works, and can also be copied and run on other devices.

There's prior art in this space — capture-and-replay isn't a new idea. The main places klura tries to push it forward:

• page-script tier — here the saved strategy is a snippet of JavaScript that the AI codes for you. klura injects it into the live, already-authenticated page and runs it there. The pure-HTTP approach (the "convert to a requests / curl script" pattern) hits a wall on sites that bind the request to in-page state — rotating per-session CSRF tokens (GitHub's X-Fetch-Nonce is one public example), request bodies built by a JS function in the bundle, binary WebSocket transports. Reproducing those from outside the browser means porting the site's own signing/encoding logic — fragile, and often impossible. Page-script sidesteps it: the saved JS calls the page's own functions — the request signer, the encoder, the socket the page already has open — so the site does the hard part and klura just collects the result. And it isn't slow the way browser automation usually is — klura keeps a warm pool of already-open, already-authenticated pages, (configurable of course) so you don't pay browser startup per call; a page-script run is dominated by the actual request and lands close to plain-curl latency.

• A real RE + debugging toolkit the agent drives — most tools capture traffic and pattern-match. Klura hands the agent an actual JS debugger: breakpoints, stepping, live-stack inspection, WebSocket frame tooling etc. It reverse-engineers a site the way a human would — break on the function that builds the request, read the encoder, verify it reproduces. You never touch any of it; you ask in plain language. Call it vibe-RE. (Detail in How the discovery works below.)

• Self-healing — when a site changes shape and a saved strategy breaks, klura re-discovers the delta and patches the strategy automatically, instead of silently returning garbage or making you re-record from scratch.

• Runs anywhere, hands off to a human when it must — klura is an MCP server, so the same saved skill is callable from Claude Code, Claude Desktop, Cursor, Windsurf, the CLI, or a programmatic import. Skills live globally at ~/.klura/skills/<platform>/, not per-project. When a site genuinely needs a human (login, 2FA, captcha), it escalates to a remote viewer — you do that one step, the agent continues, and the saved strategy runs in the resulting authenticated session.

• Plugability — The browser driver is swappable: default is Playwright, klura ships a stealth driver that fixes the standard automation-fingerprint leaks (navigator.webdriver, canvas/WebGL consistency), and you can drop in your own. The interruption layer is pluggable too — when a site throws something up mid-flow (login, captcha), the handler that decides what happens is yours to define.

The artifact

Every skill is a single JSON file on disk. A fetch-tier example for a public search API:

{
  "strategy": "fetch",
  "method": "GET",
  "baseUrl": "https://hn.algolia.com",
  "endpoint": "/api/v1/search",
  "params": {
    "query":       "{{query}}",
    "tags":        "story",
    "hitsPerPage": "{{count}}"
  },
  "response": { "format": "json", "extract": "hits[]" }
}

The LLM that produced it leaves notes fields explaining the why of each parameter, the prereq chain, and what shape the response has. Read the file → you've reverse-engineered the site.

Execution path

Once saved, run it via klura (recommended):

klura execute hackernews search_stories --args '{"query":"show hn","count":3}'

Or you could ask your LLM again, it will prefer to execute already created strategies above creating new ones.

For page-script strategies the saved JS has to run inside the live authenticated page, so that path needs the klura runtime regardless of how you wire it up. For fetch strategies the saved JSON file gives you method + URL + body/header templates — slot in your params/cookies and curl away.

Three execution tiers

Tier	What runs	When it lands	Curl-able?
fetch	Static HTTP from Node, templated body/headers + prereq chain	Plain APIs (HN search, Amazon /s?k=, GitHub GETs)	yes
page-script	JS dispatched inside the live authenticated browser tab	Sites that bind requests to in-page state	no — page state needed
recorded-path	UI action replay through the driver	Fallback when neither above is reconstructible	no

How the discovery works

There's no recorder UI implemented or planned at the moment. The agent itself drives the browser. You tell your MCP host in plain language — "search HN for the top posts this week using klura" — and that's it. No script to write, no breakpoints to set, no JS to read. You could call it vibe-RE: you describe what you want, klura performs it and figures out how the site does it. You can also ask klura to map out a site, if you just want it to observe the API without performing a specific action.

For readers who want the how — on hard sites, the agent has access to a full RE toolkit (none of which you ever touch as a user):

• JS debugger: set_breakpoint, step, resume_execution, wait_for_pause — break on the function the page calls before dispatch, read the live stack, see the encoded payload the page is about to send.
• JS source navigation: search_js_source, read_js_function, list_loaded_scripts — find the function in the bundle that builds the body or signs the request.
• WebSocket frame tooling: inspect_ws_frame, find_in_ws_frame, explain_ws_frame_structure, get_send_encoder, try_generator_in_page — for binary protocols, capture frames live and verify the captured encoder reproduces the wire format before saving.
• Network log (get_network_log) for HTTP and WS in one feed, filterable.
• Live page inspection: a11y tree, screenshots, page text, arbitrary js_eval.

This is how klura one-shots genuinely hard flows — for instance a mainstream chat app whose message-send rides a binary MQTT-style WebSocket, with an in-page codec, snowflake IDs beyond Number.MAX_SAFE_INTEGER, and a per-session token that rotates. The agent debugs the way a human would: set a breakpoint at the call site, read the actual encoder, verify it reproduces the wire output, save it as a page-script. The same toolkit generalizes to anything that builds a signed or encoded payload in-page — rotating CSRF on graphql endpoints, signed URL params, custom binary framings.

Once on disk, the next call hits the saved skill directly — or you cat the JSON and write your own scraper.

Cost: LLM once, then never

Klura runs the LLM during discovery, then never again. After LIFT the saved skill is a plain request (or a page-script dispatch): no model, no agent loop, no tokens. The only time the LLM will need to be looped back is when self-healing is required.

A couple of concrete runs, same task, measured against a plain browser agent on the same model: a Hacker News search that costs the browser agent ~20s and ~$0.09 every time runs from the saved skill in ~270ms at zero token cost. A mainstream chat app's message-send took ~27 minutes on the first run — that's the agent reverse-engineering the binary WebSocket — but every send after is a sub-millisecond dispatch into the authenticated page. The one-time discovery cost scales with site difficulty; the warm cost is always ~zero.

Install

# As an MCP server (Claude Code)
claude mcp add klura -- npx -y @klura/mcp

As an MCP server it supports many other harnesses, please check the documentation for yours to figure out how to install it.

Repo: klura · npm: @klura/runtime, @klura/mcp

Currently at 0.2.2. Real-world feedback welcome — especially the shape of the failures.

Hey guys,

I wanted to share a project I've been working on: SherlockMaps, an open-source Google Maps webcrawler built with Python and Playwright. You can check it out here.

What is it?

SherlockMaps extracts detailed company information from Google Maps searches. You give it a search term (like "restaurants berlin"), and it returns structured data including:

• Company name, category, address, phone, website
• Rating and number of reviews
• Opening hours
• Attributes (wheelchair accessibility, etc.)
• Plus Code

Key Features

• Clean OOP architecture - Well-structured with classes, dataclasses, and design patterns
• Multiple usage modes:
  • CLI tool for quick data extraction
  • Python library for integration into your own scripts
  • REST API server for headless/production use
• Multiple output formats - JSON, CSV, pretty-print
• Deduplication based on company name + website
• URL validation to filter out invalid websites
• Docker support for easy deployment
• Chrome profile persistence - Session data persists between runs
• MIT License - Fully open source

Hope you like it, I am always open to making it better 😄

Free Google search MCP that actually works.

(Demo runs Chrome visibly for clarity. Actual usage runs headless by default.)

✅ Actually works (tested 6 free MCPs, all failed)

✅ Search + URL extract in one MCP (replaces the usual search MCP + fetch MCP combo)

✅ 4 tools: `search` / `search_parallel` / `extract` / `search_extract`

✅ No API key, no proxies, no solver

✅ Auto CAPTCHA recovery (Chrome opens, human solves once, retries)

When CAPTCHA fires on any tool, a visible Chrome window opens for a human to solve. Each solve preserves the profile's reputation with Google. Built for sustainable, ethical use.

Speed (1Gbps):

- sequential: ~1.5s/q (warm)

- 4 parallel: ~2s wall

- 10 parallel: ~5s wall

Tools: 'search' / 'search_parallel' / 'extract(url)' / 'search_extract(query)'. Last one bundles search + parallel article extraction (Readability + Turndown).

Stack: TS, Playwright + stealth, Readability, Turndown. ~600 LOC.

💻 https://github.com/HarimxChoi/google-surf-mcp

📦 https://www.npmjs.com/package/google-surf-mcp

⭐ Star helps a solo dev keep maintaining.

Ask me anything about architecture, reliability, or scaling.

TL;DR Two open-source packages that tell you exactly which browser APIs a fingerprinting script touches and what it ships home. One reads the source statically, the other instruments a real browser. Same output shape, so you can diff them.

Why

If you're working anywhere near bot detection, scraping, or building a stealth browser, you eventually need to read the JS on the site. The problem is that those s are 400KB of minified, obfuscated, often-rotating code, and nobody has time to step through them by hand.

I've spent to much time going back and forth combing through minified js and I wanted one tool that would tell me, in a single pass: which APIs does this script probe, which network sinks does it fire, and which fingerprint surfaces actually leave the browser.

What

script2builtins is the static analyzer.

• Parses with acorn (module, then script fallback).
• Walks the AST, resolves aliases through string concat and variable reassignment.
• Matches every property access against a curated catalog of fingerprinting APIs across navigator, screen, canvas, WebGL, audio, WebRTC, timing, headless tells, sensors, media permissions, intl.
• Scans network sinks (fetch, XHR, sendBeacon, WebSocket, image src, script src, EventSource, Worker, navigation) and traces each body to figure out which cataloged values flow into it.
• Flags dynamic hazards (eval, Function constructor, with, document.write, computed properties) where static reach ends.

script2builtins-runtime is the dynamic companion.

• Drives a real browser session (Puppeteer or Playwright).
• Traps every catalog API, sink, and dynamic-execution point as it actually fires.
• Emits findings in the same Report shape as the static analyzer, so you can lay them side by side.

Static tells you what could be probed. Runtime confirms what was. The gap between them is where most of the interesting behavior lives. Lazy-loaded modules, environment-gated branches, you know the kind.

Live demo

Both run daily against real production loaders at https://richards.foo/tools/bot-detectors. Fresh report every 24 hours, previous hash retained so you can see what each vendor pushed overnight.

Links

• npm: script2builtins, script2builtins-runtime
• GitHub: https://github.com/StackedQueries/script2builtins / https://github.com/StackedQueries/script2builtins-runtime

Asking

Open to feedback, especially from anyone who's reverse-engineered niche detectors. What's missing from the catalog? Which vendor do you wish was covered first?

So long story short I tried scraping a website but it had otp logins . So I vibecoded the entire night into making a flutter app that will listen for messages , select the otp code using regex and forwards it to a custom backend server. So it basically solves the otp login problem if we maintain it in a continous session 🤷🏾‍♂️.

Hope it will be useful for someone

Github : https://github.com/jidukrishna/otp_listener

reCAPTCHA v3 at 0.3, FP Pro flagging bot:true, Cloudflare banning my ASN on sight. Sick of it.

Built this: a Firefox patched at the C++:

reCAPTCHA v3 0.90, FP Pro bot=false, tampering=false · CreepJS 0 lies · sannysoft all green · WebRTC no leak.

Self-hosted, MIT, no cloud, no subscription.

Repo: https://github.com/P0st3rw-max/stealthfox

Hello everyone,

It has been some time since I wrote here :)

A lot has happened since my last post, and we are now almost at 1.5M downloads.

I wanted to share this update with you here, as many people have asked for these features since the first release.

Introducing the new spiders templates feature to make generic spiders easier. Now you have a CrawlSpider and a SitemapSpider. Both offer many options and controls. We also added tools to make it easier, like LinkExtractor and CrawlRule, which will help you build your generic spider. Stop rewriting "follow links matching X" boilerplate in every parse() :D

Check the new page for them on the website here: https://scrapling.readthedocs.io/en/latest/spiders/generic-templates.html

Also, we have improved the adaptive feature so it's now much better, fixed some bugs, improved the docs, and updated browsers/fingerprints

The full release notes are here: https://github.com/D4Vinci/Scrapling/releases/tag/v0.4.8

Let me know your thoughts and what I should add next :)