u/ShilpaMitra

Two very different repos landed that are worth knowing if you build with agents. One is the memory layer (a vector database that runs inside your process, no server), the other is the supervision layer (a runtime that records an agent's run as a reversible trace so nothing touches your files until you accept it). One is battle-tested and popular, the other is a promising research alpha. Here is what each does, the exact install, and the honest catch.

zvec: an in-process vector database, the SQLite of vector search

Stars / Status / License: ~13.3k, actively maintained (v0.4.0, May 2026), Apache 2.0.
Repo: https://github.com/alibaba/zvec

zvec is a vector database that embeds directly into your app, no separate server to run, config, or babysit. Alibaba built it and runs it internally, and the pitch is the same one SQLite makes for relational data: for a lot of workloads you do not need a database server, you need a fast library that lives in your process. It does dense and sparse vectors, hybrid search (similarity plus structured filters in one query), and it uses write-ahead logging so your data survives a crash.

Install and a full working example, straight from the README:

pip install zvec


import zvec

schema = zvec.CollectionSchema(
    name="example",
    vectors=zvec.VectorSchema("embedding", zvec.DataType.VECTOR_FP32, 4),
)
collection = zvec.create_and_open(path="./zvec_example", schema=schema)

collection.insert([
    zvec.Doc(id="doc_1", vectors={"embedding": [0.1, 0.2, 0.3, 0.4]}),
    zvec.Doc(id="doc_2", vectors={"embedding": [0.2, 0.3, 0.4, 0.1]}),
])

results = collection.query(
    zvec.VectorQuery("embedding", vector=[0.4, 0.3, 0.3, 0.1]),
    topk=10,
)
print(results)

There is a Node.js package too (npm install u/zvec/zvec) and a Dart/Flutter SDK. Supported platforms are Linux (x86_64 and ARM64), macOS (ARM64), and Windows (x86_64). Two heads-ups from actually running this: create_and_open wants a path that does not already exist (point it at a fresh directory, not a pre-made temp dir), and newer zvec versions deprecate VectorQuery in favor of Query, though VectorQuery still works for now.

The catch: in-process is the strength and the limit. Multiple processes can read a collection at once, but writes are single-process exclusive, so there is one writer at a time and no built-in clustering. This is the right tool for embedding search into an app, a notebook, a CLI, or an edge device, and the wrong tool if you need a horizontally scaled, many-writer database service, where a server like Milvus or Qdrant still fits better. Also note the headline "billions of vectors in milliseconds" is Alibaba's own benchmark, so measure it on your data and your hardware before you quote it. macOS is ARM64 only on the current build, so no Intel Macs.

→ The verified in-process vector search setup, actually run in CI

Shepherd: a reversible, Git-like trace for agent runs

Stars / Status / License: ~742, early alpha, MIT.
Repo: https://github.com/shepherd-agents/shepherd

Shepherd is a runtime substrate for agent work that needs inspection, reversibility, and supervision. The core idea: an agent's task comes back as a reviewable proposal, not a live edit. Nothing touches your files until you accept it. The run is recorded as a durable, inspectable trace you can observe, replay, and revert, and the agent's output is held to one side as a retained output you can run and inspect before you keep or discard it.

The part I found most clever is that permissions live in the function signature. A task is a plain Python function whose signature declares a read-only or read-write grant per repository, and on a supported OS that grant is enforced at the native syscall jail (macOS Seatbelt, Linux Landlock). A write to a read-only repo is refused by the operating system itself, before any undo point, not merely caught at a merge gate.

Install and the keyless offline quickstart, from the README:

pip install shepherd-ai


shepherd init                        # make this directory a Shepherd workspace
shepherd demo write quickstart > quickstart_demo.py
python quickstart_demo.py            # run a task; its result is retained, not applied
shepherd run changeset --latest      # see what it wrote, held to one side
shepherd run select <run-ref>        # keep it   (or: shepherd run discard <run-ref>)

There is a live agent lane too, where the body of a task is a Claude agent (it needs the claude CLI signed in or an ANTHROPIC_API_KEY), but the offline lane above runs anywhere with no key.

The catch: this is early alpha and the maintainers say so, with APIs that may change between releases, and at ~742 stars it is a young research project (there is a paper, arXiv 2605.10913, with authors including Christopher Manning). Treat it as a promising experiment to learn from, not a dependency to ship on yet. The strong sandbox guarantee is also platform-dependent: enforcement is exercised on macOS Seatbelt, while Linux Landlock is container-gated, so the syscall-level protection is not uniform across machines. And the "~5x faster than docker commit, ~95% KV-cache reuse" figures are the authors' own benchmarks, so read them as claims to reproduce, not settled facts.

→ The verified least-privilege grants setup

Who each is for

If you are building retrieval or memory into an app and do not want to run a vector server, reach for zvec today: it is mature enough, popular, and Apache-licensed. If you are building or operating agents and want a real undo button plus OS-level permission enforcement, Shepherd is worth studying now and piloting in a sandbox, but keep it out of production until it leaves alpha. Together they sketch a nice pattern: zvec as the memory an agent reads from, Shepherd as the safe hands it acts with.

We keep a running, machine-verified collection of workflows built on tools like these in the open: https://github.com/Neeeophytee/awesome-ai-workflows

The essay making the rounds this week is from Thariq Shihipar on the Claude Code team: the map is not the territory. Your prompt is a map, the codebase is the territory, and the gap is your unknowns. His claim is that Fable 5 is the first model where output quality is bottlenecked by your ability to clarify those unknowns, not by the model.

The essay describes eight working techniques. I distilled them into installable skills on the agentskills.io standard, so instead of remembering the patterns, you just invoke them:

• blindspot-pass: surface your unknown unknowns in an unfamiliar area before you prompt
• brainstorm-prototypes: wildly different throwaway variations to react to, for taste you cannot verbalize
• interview-me: one question at a time, architecture-changing questions first
• reference-hunt: point at working source code as the spec, even across languages
• implementation-plan:a plan that leads with the decisions you will most likely tweak
• implementation-notes: log every deviation so the next attempt learns from this one
• pitch-packager: bundle spec + prototype + notes into a buy-in doc, demo first
• change-quiz: a comprehension quiz you must pass before you merge

Install in Claude Code (tested, works):

/plugin marketplace add Neeeophytee/finding-unknowns-skills
/plugin install finding-unknowns@finding-unknowns-skills

Or copy any single skills/<name>/ folder into .claude/skills/, or drop the one-file CLAUDE.md into your project as passive guidance.
The repo's EXAMPLES.md has ready-to-paste example prompts for every skill.

Repo: https://github.com/Neeeophytee/finding-unknowns-skills
License: MIT.

Which of the eight do you actually need most? My bet is most people skip the interview and pay for it during implementation.

A big star count tells you a repo is popular. It does not tell you the license is homemade, the name is misleading, or the tool is one missing consent form away from fraud. Here are five useful open-source AI projects, each with what it nails and the catch the hype skips. Two have license or naming fine print worth knowing before you build on them, and two are dual-use in ways the pitch does not mention. Star counts are approximate and drift.

1. MinerU: turn any PDF or office doc into clean, LLM-ready markdown

Stars / Status / License: ~70k, actively maintained, custom "MinerU Open Source License" (see catch). Repo: https://github.com/opendatalab/MinerU

This is the fix for garbage RAG inputs. MinerU (from OpenDataLab) parses PDFs, DOCX, PPTX, XLSX, and images into clean markdown and JSON, with strong OCR, so your retrieval pipeline is not choking on broken tables and jumbled columns. If your RAG answers are bad, the cause is usually the input, and this is where you fix it.

The catch: the license is not plain Apache or MIT. MinerU moved off AGPLv3 to its own "MinerU Open Source License," which is based on Apache 2.0 but adds conditions. It is fine for most uses, but read the terms before you build a commercial product on it rather than assuming permissive defaults.

2. voicebox: a local, open-source voice studio

Stars / Status / License: ~35k, actively maintained, open-source (confirm the exact license on the repo). Repo: https://github.com/jamiepine/voicebox

From Jamie Pine (of Spacedrive), voicebox is a local-first alternative to ElevenLabs: clone a voice from a few seconds of audio, generate speech across many languages and TTS engines, and dictate into any text field with a global hotkey, with transcription running on local Whisper. The whole voice stack runs on your machine, which is the real draw for privacy and cost.

The catch: voice cloning is dual-use. Cloning a voice you do not have permission to use is how impersonation and fraud happen, so treat consent as a hard requirement, not a nicety, and clone only voices you are actually allowed to.

3. ai-website-cloner-template: reverse-engineer a site with your coding agent

Stars / Status / License: ~25k, actively maintained, open-source (confirm the license on the repo). Repo: https://github.com/JCodesMore/ai-website-cloner-template

Point your AI coding agent at a URL, run one command, and it inspects the site, extracts design tokens and assets, writes component specs, and rebuilds the thing as a clean Next.js plus shadcn/ui codebase. As a way to learn how a well-built site is structured, or to prototype fast, it is a clever use of a coding agent.

The catch: this one is legally and ethically loaded. Cloning someone else's live site copies their design and content, which is an intellectual-property problem, and the exact same capability is how phishing and spoof sites get built. Use it on your own sites, on sites you have permission to copy, or as a learning exercise you do not ship. Do not pass off a clone of someone else's site as your own.

4. Anthropic-Cybersecurity-Skills: a big library of security skills for agents

Stars / Status / License: ~24k, actively maintained, Apache 2.0. Repo: https://github.com/mukul975/Anthropic-Cybersecurity-Skills

First, a correction the name invites: despite "Anthropic" in the title, this is a community project by an independent developer, not an official Anthropic repo. With that clear, it is a large, well-organized set of 800-plus cybersecurity skills for AI coding agents, each mapped to standard frameworks (MITRE ATT&CK, NIST CSF, D3FEND, and others) and usable across Claude Code, Copilot, Cursor, and 20-plus other agents. For security teams, having agent skills tied to recognized frameworks is a real convenience.

The catch: this is dual-use security tooling, spanning both defensive and offensive frameworks. It is meant for security professionals, and running offensive techniques against systems you do not own is illegal. Review any skill before you let an agent execute it, and keep this firmly in a blue-team, authorized-testing, or lab context.

5. agent-native: build apps for agents from day one

Stars / Status / License: ~3.4k, new in 2026, open-source. Repo: https://github.com/BuilderIO/agent-native

From Builder.io, a framework for apps where the agent and the UI share the same actions and state from the start, rather than bolting an agent on afterward. Its architecture rules are opinionated (data in SQL, all AI routed through the agent, operations as shared actions, UI and agent kept live-synced), and it ships 15-plus cloneable SaaS templates to start from.

The catch: it is new and small, so this is an early, interesting bet rather than a proven standard. The ideas are worth studying even if you do not adopt the framework, but treat production use as experimental for now.

How to pick if you only try one

If your AI outputs are bad, start with MinerU, because clean inputs fix more problems than any prompt tweak. If you want a private voice stack, voicebox. If you are learning front-end or prototyping, the website cloner. Security teams get real mileage from the skills library with the caveats above, and agent-native is the one to read about now and maybe build on later.

Checking the fine print before you build is a habit worth automating, so we turned it into a verified recipe: it makes you record each dependency's real license (no assuming permissive), flag any non-standard license as reviewed, and attach an authorization gate to every dual-use tool, using these exact five repos as the worked example. Vet the fine print before you build.

We keep a running, machine-verified collection of workflows built on tools like these in the open: https://github.com/Neeeophytee/awesome-ai-workflows

OpenRouter shows, per app, which models each agent burned tokens on. I pulled the "this month" breakdown for four of the biggest (Hermes Agent, OpenClaw, Claude Code, and Kilo Code) and the pattern is the same everywhere except one telling exception. All numbers below are from OpenRouter's public per-app model pages for the current month.

The top of each agent's list

Hermes Agent (this month)
1  Owl Alpha          openrouter   6.95T
2  DeepSeek V4 Flash  deepseek     5.02T
3  MiniMax M3         minimax      2.99T
4  Step 3.7 Flash     stepfun      2.24T
5  DeepSeek V4 Pro    deepseek     1.55T
   first Anthropic model: Claude Opus 4.8 at #8 (652B)


OpenClaw (this month)
1  MiniMax M3         minimax      1.34T
2  DeepSeek V4 Flash  deepseek     472B
3  Owl Alpha          openrouter   377B
4  Nemotron 3 Super   nvidia       294B
5  Claude Sonnet 4.6  anthropic    267B
   first Anthropic model: Claude Sonnet 4.6 at #5


Kilo Code (this month)
1  Laguna M.1         poolside     1.93T
2  Step 3.7 Flash     stepfun      1.38T
3  MiniMax M3         minimax      880B
4  Owl Alpha          openrouter   612B
5  Nex-N2-Pro         nex-agi      561B
   first Anthropic model: none in the top 10


Claude Code (this month)
1  Claude Opus 4.8    anthropic    915B
2  Owl Alpha          openrouter   455B
3  Claude Sonnet 4.6  anthropic    406B
4  GLM 5.2            z-ai         306B
5  DeepSeek V4 Flash  deepseek     289B
   this is the only one where Anthropic tops the list

The pattern

Add up each agent's top ten and split it into Anthropic's Claude models versus everything else (open, cheap, or specialized):

Agent          Open / cheap    Anthropic (Claude)
Kilo Code          100%              0%
Hermes Agent        95%              5%
OpenClaw            84%             16%
Claude Code         49%             51%

Three of the four agents send the overwhelming majority of their tokens to cheap or open models: DeepSeek, MiniMax, Step, Laguna, Nemotron, GLM, MiMo, and OpenRouter's own Owl Alpha slot. Kilo Code does not run a single Claude model in its top ten. The one exception is Claude Code, Anthropic's own tool, and even it pulls in GLM 5.2, DeepSeek, MiniMax, and MiMo alongside the Claude models. If token volume were the scoreboard, the frontier labs would look like they had already lost.

The flip: volume is not value

They have not lost, and here is why the chart misleads. Tokens measure volume, not spend or quality. Cheap open models rack up enormous token counts because agents pour bulk work, tool calls, retries, and long contexts through them, where price per token is what matters. Premium models get reached for the fewer, harder calls, and they cost far more per token. Platform-wide, Anthropic is only about 12% of OpenRouter's tokens but roughly 46% of its revenue. A model can sit at #8 by tokens and still dominate the bill.

We turned that exact reversal into a verified recipe: it attributes the same usage by tokens and by dollars and proves the flip, cheap-open winning volume while premium wins cost. Token volume versus cost receipts.

How to read this without being fooled

These are per-app figures from OpenRouter only, so they capture just the traffic each agent routes through OpenRouter. That matters most for Claude Code, which sends a large share of its work straight to Anthropic's own API, so its true Claude lean is higher than the 51% here, and this view undercounts it. Read the other three as close to their real diets and Claude Code as a partial slice.

Owl Alpha shows up across every agent under OpenRouter's own slot. It is a routed or preview model listed by the platform rather than a named lab model, so treat its rank as "whatever OpenRouter is currently steering there," not a specific product.

Where this connects to what we do

We keep a running, machine-verified collection of the agent and routing workflows behind these numbers in the open: https://github.com/Neeeophytee/awesome-ai-workflows

GLM-5.2 is a strong, cheap, open-weights model, and Hermes Agent treats Z.AI as a first-class provider, so wiring the two together is a two-line job. Here are the three routes that actually work (direct Z.AI, OpenRouter, and the flat-rate GLM Coding Plan), the verbatim commands for each, and the honest catches. Pick one.

One-time setup

Install Hermes and reload your shell:

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
source ~/.bashrc   # or source ~/.zshrc

Hermes stores secrets in ~/.hermes/.env and non-secret settings in ~/.hermes/config.yaml. The hermes config set command puts each value in the right place for you.

Hermes Agent + GLM-5.2

Stars / Status / License: Hermes ~204k stars, MIT.
GLM-5.2: Z.ai, MIT open weights, 1M-token context.
Repos: https://github.com/NousResearch/hermes-agent and
model page https://openrouter.ai/z-ai/glm-5.2

Route A: direct from Z.AI (recommended)

Z.AI is a dedicated Hermes provider with the id zai, so you just set the key and pick the model:

hermes config set GLM_API_KEY <your-z.ai-key>
hermes config set model zai/glm-5.2
hermes

That is it. One useful detail from the docs: when you use the Z.AI provider, Hermes automatically probes the global, China, and coding-plan endpoints to find the one your key accepts, so you normally do not set a base URL by hand. If you prefer the interactive route, run hermes model, choose the Z.AI / GLM provider, and enter glm-5.2. A one-shot test without changing your default:

hermes chat --provider zai --model glm-5.2

Route B: through OpenRouter (easiest to compare and often cheapest)

If you want one key that also reaches every other model, go through OpenRouter and point Hermes at the GLM-5.2 slug:

hermes config set OPENROUTER_API_KEY <your-openrouter-key>
hermes model     # choose OpenRouter, then enter the model: z-ai/glm-5.2

OpenRouter routes your request to whichever host is cheapest or fastest, which makes it the simplest way to A/B GLM-5.2 against Opus or anything else on your own prompts.

Route C: the flat-rate GLM Coding Plan

If you code all day and want a flat monthly bill instead of a per-token, use a GLM Coding Plan key. It is still a GLM_API_KEY to Hermes, and the same auto-detection finds the coding endpoint, so the setup is identical to Route A:

hermes config set GLM_API_KEY <your-coding-plan-key>
hermes config set model zai/glm-5.2

If you ever need to pin the coding endpoint explicitly (for example, to be sure you are not hitting the general one), set the base URL:

hermes config set GLM_BASE_URL https://api.z.ai/api/coding/paas/v4

Verify it actually works

Start a chat and confirm the banner shows your model, then ask something specific:

hermes

If it errors, hermes doctor diagnoses config problems, and hermes model lets you re-pick the provider and model. Do not move on to the gateway or automations until a plain chat works on GLM-5.2.

A nice bonus: use it as a Mixture-of-Agents reference

Because GLM-5.2 is now a normal model in Hermes, you can also drop it into an MoA preset as a cheap reference model that feeds a stronger aggregator, getting a second perspective on hard turns without paying frontier prices for every token. That is optional, but it is one of the better uses of a cheap, capable model inside Hermes.

How to pick your route

Use Route A (direct Z.AI) for the canonical model and predictable behavior. Use Route B (OpenRouter) if you want one key for everything and the cheapest hosting, or if you are comparing models. Use Route C (Coding Plan) if your usage is heavy and steady enough that a flat monthly rate beats a per-token. All three land you on the same glm-5.2.

The catches (and the soundness notes)

Hermes requires a model with at least 64k context, per its docs. GLM-5.2's 1M context clears that with room to spare, so this is a non-issue here, but it is why some smaller models get rejected.

The cheapest OpenRouter hosts sometimes serve a quantized GLM-5.2, which is very good but not identical to the full-precision model, so if output quality matters, check which host you landed on or prefer the direct Z.AI route.

Watch two GLM-5.2 details regardless of route: several hosts cap output at around 32,768 tokens per call despite the huge input context, so long generations come back in chunks, and the GLM Coding Plan's headline low prices are promotional intro rates that step up in later cycles, so price it on the standing tier.

And a governance note: GLM-5.2 is a Chinese-lab model served from multiple regions. Hermes auto-detects the global versus China endpoint, so if data residency matters to you, pin GLM_BASE_URL to the endpoint you actually want rather than letting auto-detection choose.

→ The verified setup, with CI proof & readymade prompt

If you want the wider view on getting near-frontier quality without frontier bills, this companion piece covers two more ways to do exactly that.

"Agent memory" gets used as one word, but it is really several different mechanisms doing different jobs. Here are seven types, and for each one an open-source project that actually implements it plus the kind of workflow that leans on it. Two honest framings up front: this seven-way split is a useful lens, not a hard standard (different sources slice memory differently), and most real agents use only two or three of these, not all seven. Star counts below are approximate and drift; licenses are noted where confirmed and flagged where you should check.

1. In-context (working) memory

What it is: whatever is in the model's context window right now, the equivalent of what you are holding in your head this second.

Repo: letta-ai/letta (Apache-2.0), the MemGPT project. It treats the context window like RAM, keeping a small core in-context and paging older material out to searchable storage.

The workflow: any long chat or coding session where the agent has to track the current task, the files it just touched, and the last few steps without re-reading everything. Letta's job is deciding what stays in the window and what gets summarized out.

The catch: context is finite and every token costs money and latency. "Unbounded context" here means managed forgetting (summarize and page out), not true unlimited recall, so detail is lost at the edges. Plan for that, do not assume perfect memory of the whole conversation.

Verified recipe: letta-agent-managed-memory

2. Semantic memory

What it is: durable, general facts, your preferences, your stack, who is who, definitions, decoupled from when you learned them.

Repo: topoteretes/cognee (open-source, check the repo for the exact license), which builds a self-hosted knowledge graph plus vector index so facts are both searchable by meaning and connected by relationships.

The workflow: a personal or team assistant that should stop asking the same questions. "My database is Postgres, my framework is FastAPI" gets stored once and recalled forever.

The catch: the graph is built by an LLM extracting facts, so it inherits extraction errors and can store something confidently wrong. A knowledge graph is also real maintenance, not a free lunch, so treat recalled facts as strong hints, not gospel.

Verified recipe: cognee-knowledge-graph-memory

3. Episodic memory

What it is: specific past events with a time to them, "last Tuesday you decided X," rather than timeless facts.

Repo: getzep/graphiti (open-source, verify the license on the repo), a temporally-aware knowledge graph that ingests interactions as dated episodes and supports point-in-time queries.

The workflow: a long-running support or personal agent that needs to answer "what happened, and when." It is what lets an assistant reference a decision from three weeks ago correctly instead of blending it with today.

The catch: a temporal knowledge graph is heavy infrastructure (a graph database and an ingestion pipeline), and recall is only as good as how cleanly episodes were captured. For a simple app this is overkill; reach for it when the timeline truly matters.

Verified recipe: graphiti-temporal-graph-memory

4. Procedural memory

What it is: learned how-to, reusable skills the agent builds from doing tasks, not facts it looked up.

Repo: MineDojo/Voyager (MIT), the classic example. When a task succeeds, the working code is saved to a skill library indexed by a description, and relevant skills are retrieved and reused later.

The workflow: automation that should get better with practice, a deploy runbook, a scraping routine, a data-cleanup script that the agent captures once and reruns. (This is the same idea behind Hermes Agent's /learn skills.)

The catch: Voyager is a research project (it lives in Minecraft), so treat it as the concept demo, not a drop-in production library. And a saved skill can be over-fit to the one run that produced it or subtly wrong, so procedural memory needs review before you trust it, exactly like a script you did not write.

Verified recipe: voyager-skill-library-pattern

5. External / retrieval memory

What it is: knowledge kept outside the model and pulled in on demand, the classic RAG pattern.

Repo: run-llama/llama_index (MIT), the leading framework for indexing your documents and retrieving the relevant pieces at query time.

The workflow: a chatbot or assistant that answers from your own corpus, "answer using these 500 PDFs," where the knowledge is too big or too changeable to bake into the model.

The catch: retrieval memory inherits every RAG failure mode. If your chunking, indexing, or query is off, it recalls something stale, irrelevant, or nothing at all, and then answers anyway. The memory is only as good as the index behind it.

Verified recipe: llamaindex-retrieval-memory-rag

6. Parametric memory

What it is: knowledge stored in the model's own weights, what it "just knows" without being told, shaped by training and fine-tuning.

Repo: unslothai/unsloth (Apache-2.0 core; the Unsloth Studio UI is AGPL-3.0, so mind the copyleft if you build on it). It is one of the most efficient ways to fine-tune a model, which is how you write to parametric memory.

The workflow: baking your domain, jargon, or house style into a small model so it is always on without spending prompt tokens or a retrieval call. Fine-tune once, and the knowledge is native.

The catch: parametric memory is expensive to write and static once written. Updating it means retraining, and fine-tuning risks overfitting and catastrophic forgetting (gaining your domain while losing general skill). Use it for stable, always-needed knowledge, not for facts that change weekly.

Verified recipe: unsloth-parametric-finetune-config

7. Prospective memory

What it is: remembering to do something in the future, follow-ups, reminders, scheduled steps, rather than recalling the past.

Repo: agentscope-ai/ReMe (open-source, check the repo for the license), which consolidates conversations into memories and surfaces proactive reminders on a schedule.

The workflow: an agent that says "I will check the deploy in an hour" or "send the weekly digest every Monday" and actually does. This is the memory behind scheduled and triggered actions.

The catch: this is the least standardized of the seven, and most implementations are really a task queue or cron plus a stored list of intentions, not a distinct memory engine. The repo here is newer and smaller, so treat it as a pattern to copy more than a proven dependency.

Verified recipe: reme-prospective-schedule

Which ones you actually need

Do not try to build all seven. Most agents need working memory (unavoidable) plus one or two others chosen by the job: retrieval memory if the knowledge is big and changeable, semantic or episodic if it is a long-lived personal assistant, procedural if the agent should learn repeatable tasks, parametric only when the knowledge is stable enough to be worth training in, and prospective the moment your agent needs to act on a schedule. Pick by the workflow, not by the taxonomy.

If you want to see all seven implemented in runnable code, this collection is a solid hands-on reference: NirDiamant/Agent_Memory_Techniques.
We also keep our own verified agent and memory workflows in the open here: https://github.com/Neeeophytee/awesome-ai-workflows

Three jobs, three places where a much cheaper model does nearly the same work as the big name. For bulk text, Xiaomi's open-weights MiMo V2.5 stands in for OpenAI's small model. For images, Alibaba's Wan 2.5 takes on GPT-Image-2. For video, Kuaishou's Kling 3.0 takes on Sora 2. Each is the right default for most of its category, and each saves real money, between six and sixteen times depending on the job.

The part to read carefully is the "only a few percent worse" framing. Those gaps are leaderboard and vendor figures, and a single percentage hides the specific tasks where the premium model still wins outright, which for one of these three matters far more than the chart admits. Prices were checked recently and drift, so treat them as current-ish. Real numbers and honest caveats below.

Bulk text: swap GPT-5.4 mini for MiMo V2.5

Model                    Input /1M     Output /1M
MiMo V2.5 (Xiaomi)       $0.105        $0.28
MiMo V2.5 Pro            $0.435        $0.87
GPT-5.4 mini (OpenAI)    $0.75         $4.50

MiMo V2.5 is Xiaomi's open-weights model, and on output tokens the base version is roughly 12 to 16 times cheaper than GPT-5.4 mini. The quality is close on the things bulk work actually needs: the stronger MiMo V2.5 Pro lands around 57% on SWE-bench Pro, within about a point of GPT-5.4, and Xiaomi reports it uses meaningfully fewer tokens to get there.

The catch: do not mix the variants. The headline "12x cheaper" is the base V2.5, and the headline "basically as good" is the Pro variant, which costs more (still cheaper than the OpenAI mini, but not 12x). Pick base V2.5 for high-volume, low-stakes work, where a small quality gap does not matter and the price difference does. Keep a premium model for the small slice of work where one wrong answer is expensive.

Image generation: swap GPT-Image-2 for Wan 2.5

Model            Price per image (1024x1024)
Wan 2.5          ~$0.03
GPT-Image-2      $0.006 low / $0.053 medium / $0.211 high

Against high-quality GPT-Image-2, Wan 2.5 (Alibaba, API-only) is roughly 7 to 8 times cheaper per image. For general scenes and aesthetics, the gap a normal viewer notices is small.

This is the one where "about 5% worse" is misleading, so here is the honest version. GPT-Image-2 currently sits at the top of the Artificial Analysis image leaderboard with the largest first-to-second lead that board has recorded, and it dominates specifically on text inside images, dense layouts, infographics, slides, and multilingual typography. So if your images are mostly pictures, Wan is a great, cheap swap. If your images contain words, charts, or precise layout, GPT-Image-2 is not "5% better," it is in a different class, and Wan will frustrate you. Match the tool to whether text is in the frame.

Video: swap Sora 2 for Kling 3.0

Model        Price per second of video
Kling 3.0    ~$0.10  (roughly $0.09 to $0.14)
Sora 2       ~$0.75

This is the cleanest swap of the three. Kling 3.0 is about 6 to 7 times cheaper per second, and "roughly equal" is fair overall, with a twist: they lead on different things. Kling 3.0 outputs native 4K and is excellent at human motion (dancing, martial arts, running without limbs melting). Sora 2 caps standard output lower but leads on world physics and longer, coherent storytelling. For most short clips, social content, and concept iteration, Kling at a sixth of the price is the obvious default. For physics-heavy or film-grade final shots, Sora 2 still earns its premium. A common pattern is Kling for the many rough iterations, Sora for the one hero shot.

How to pick

The honest rule across all three: the cheap model is the right default for volume and iteration, and the premium model is worth its price only for the specific thing it dominates. That is MiMo for bulk text and a strong model for the high-stakes few, Wan for picture-images and GPT-Image-2 when there is text in the frame, Kling for most clips and Sora for the physics-heavy hero shot.

One of these three ships has open weights: MiMo. So if you run your own hardware, MiMo is the swap whose gap to free shrinks even further. Wan 2.5 launched API-only, and Kling is proprietary and API-only too, but both are cheap enough that it rarely matters.

We turned the image swap into a verified recipe that bakes in exactly that rule: CI proves every text-in-frame or chart-and-layout case routes to GPT-Image-2 and the plain-picture cases route to cheap Wan, so the cost saving never quietly wrecks a slide. The same cheap-iterations, premium-hero-shot pattern applies to the video swap.

→ The verified image swap, with the text-in-frame guard CI-checked

If the broader theme of getting near-frontier results without frontier prices is your thing, this companion piece covers two more ways to do exactly that.

You may have seen the claim going around that Apple "turned 2 billion iPhones into local AI machines" and that you can now export any Hugging Face model and run it natively on iPhone. The repo is real and useful. The framing is not. Here is the accurate version, with the exact commands and the honest limits, so you do not show up to your Mac expecting Llama 70B on your phone.

What it actually is

Stars / Status / License: ~1.2k stars (still new, climbing fast, and the real credibility is that it is an official Apple repo, not just community traction), BSD-3-Clause.

Repo: https://github.com/apple/coreai-models

apple/coreai-models is the open-source companion to Apple's new Core AI framework (shown at WWDC26). It is three things plus a bonus: export recipes that convert a curated catalog of popular open-source models from Hugging Face into Apple's on-device .aimodel format, Python primitives for authoring your own PyTorch models for on-device, a Swift package to run those models inside a macOS or iOS app, and a set of agent skills that teach a coding assistant how to use Core AI properly. For context, this is not Apple's first move here: coremltools and Core ML have existed for years, and Hugging Face has shipped its own exporters. This is the next step, an Apple-official, end-to-end export-plus-runtime path tied to Core AI.

Setup

You need a Mac on the new toolchain. Per the repo, the requirements are macOS and iOS 27.0+ and Xcode 27.0+. Then install uv and list what is actually supported:

brew install uv
git clone https://github.com/apple/coreai-models.git && cd coreai-models
uv run coreai.model.registry --list-models

Each model has its own export recipe in the models/ folder. Exported models come out as standalone .aimodel files you integrate through the Core AI framework, and there are CLI tools to run an exported model directly on a Mac.

The agent skills install as a plugin. For Claude Code:

/plugin marketplace add git@github.com:apple/coreai-models.git
/plugin install coreai-skills@coreai-models

There are equivalent commands for Codex CLI and Gemini CLI in the README.

The useful part

The ready-made recipes are the real story, not the hype. If you build apps, the path from "a model on Hugging Face" to "a private, offline feature in my iOS app" used to be a research project. Here it is a documented recipe plus a Swift package. The three bundled skills are scoped sensibly too: working-with-coreai (the full export-then-run workflow), model-authoring (rules for writing PyTorch that survives on-device, KV cache patterns, precision, MoE), and model-compression-exploration (systematically trying quantization and palettization). That last skill is the tell for what this is really about: making models small enough to fit on a device.

The catch (and the soundness caveats)

"Any Hugging Face model on your iPhone" is not true. This is a curated gallery of supported models with tested recipes, not a universal converter, and Apple says plainly it is a curated, well-tested set. Use --list-models to see what is actually covered before you plan around a specific model.

"2 billion iPhones" is not true either. It requires iOS and macOS 27.0+ and Xcode 27.0+, so it is the newest devices on the newest OS, not the global install base, and not older hardware. Most phones in the world cannot run this today.

On-device means small models, and that is a hard physical limit, not a tuning detail. A phone has a handful of gigabytes of memory, so the realistic candidates are small or heavily quantized models, which is exactly why a whole skill here is about compression. The "zero cloud" part is real and is the actual win: private, offline inference. Just calibrate it to small-model capability, not frontier-model capability.

And it is brand new. One commit, days old, and Apple is explicitly not accepting code contributions right now (open PRs get closed), though issues for bugs and model requests are open. Treat it as an official but early release, not a battle-tested standard yet.

→ The verified setup, with CI proof & readymade prompt

How to pick if you only try one thing

If you have a Mac on the new toolchain, run --list-models, pick a small supported model, and walk the working-with-coreai skill end to end once. That single pass teaches you more about what is realistic on-device than any benchmark thread. If you are not on macOS 27 yet, there is nothing to try here today, and that is worth knowing before you spend an evening on it.

GLM-5.2 is the model people keep saying is "almost Opus for a fraction of the price." That is half true, and the half that is true is worth a lot of money.
This is the practical guide: every cloud route to run it, the real per-token prices next to Claude Opus 4.8, and an honest read on where it matches Opus and where it does not. We covered running it locally in a separate post, so this one is all about hosted access.

What it is, in one line

Maker / License / Context: Z.ai (formerly Zhipu AI), MIT open weights, 1M-token context. Model page: https://openrouter.ai/z-ai/glm-5.2

GLM-5.2 is a 754B-parameter open-weights model from Z.ai, released mid-June 2026, built for long-horizon coding and agentic work. Because the weights are MIT licensed, it is hosted in a lot of places, which is exactly why the price floor is so low. One spec worth flagging up front: the 1M figure is the input context. On OpenRouter the model still caps output at around 32,768 tokens per call, so a giant codebase fits in the prompt, but you generate it back in chunks, not one 200-page response.

Where to run it (hosted)

The direct API (Z.ai). The first-party route. OpenAI-compatible, so most SDKs work by swapping the base URL and key. Best if you want the canonical version and predictable behavior.

OpenRouter. One key, and it routes your request to whichever of the 13-plus providers serving GLM-5.2 is cheapest or fastest. This is usually the cheapest per-token path and the easiest way to A/B it against other models without new accounts. It is also OpenAI-compatible.

Other hosts (Fireworks, DeepInfra, and similar). Because the weights are open, independent hosts serve it too, sometimes on quantized variants that shave the price further. Worth knowing the cheapest ones run a quantized model, so treat their output as very-good-not-identical to the full-precision version.

The GLM Coding Plan (subscription). Z.ai's flat-rate plan aimed at agentic coding. It ships an OpenAI-compatible endpoint that drops into Claude Code, Cline, Roo Code, Kilo Code, OpenCode, Cursor, and 20-plus other tools. This is the route to pick if you code all day and would rather pay a flat monthly fee than watch a token meter.
One honest note on the pricing you may have seen: the headline "a few dollars a month" figures from launch week were promotional intro rates. The standing tiers are higher (Lite around $18 a month, Pro around $72, Max around $160), with promo pricing that steps up in later cycles, so check the current rate before you commit.

What it costs versus Opus 4.8

Per-token list prices (approximate, and the cheapest hosted routes vary):

Route                              Input /1M     Output /1M
GLM-5.2, direct from Z.ai          $1.40         $4.40   (cached input ~$0.26)
GLM-5.2, cheapest via OpenRouter   ~$0.95-1.20   ~$3.00-4.20
Claude Opus 4.8                    $5.00         $25.00

The gap that matters is output tokens, where Opus is roughly five to six times the price. For output-heavy work (long code generation, big document drafting), GLM-5.2 is dramatically cheaper for the same volume.
The catch on cost is below: cheaper per token is not the same as cheaper per finished task, if the cheaper model needs more turns to get there.

How close is it to Opus 4.8, really

Here is the part people oversell in both directions. On a provisional third-party leaderboard (BenchLM), Opus 4.8 still leads overall, but narrowly, and the picture flips depending on the task:

Area / benchmark                 GLM-5.2     Opus 4.8
Overall (provisional index)      91          93
General coding (avg)             62.1        76.4
Long-horizon coding              74.4        75.1
SWE-Marathon (ultra-long)        26.0        13.0
Agentic (avg)                    81.0        80.1

Read that carefully. Opus is clearly ahead in general coding. The two are within a point on long-horizon coding. GLM-5.2 actually wins on the ultra-long-horizon SWE-Marathon and edges Opus on agentic average and on math (AIME 2026). So "almost Opus" is fair for long, agentic, project-scale work, and an overstatement for everyday coding, where Opus is still meaningfully better.

And GLM-5.2 is a model from a Chinese lab, served by various hosts. The open MIT weights are truly unrestricted, but if you have data-residency or governance constraints, the hosted route and the provider you pick matter, so check where your prompts actually land.

How to pick if you only try one route

If you mostly want to experiment and compare, start on OpenRouter: one key, cheapest routing, and you can pit GLM-5.2 against Opus on your own prompts in an afternoon. If you live in a coding agent all day, price out the GLM Coding Plan against your current token spend, but use the standing tier price, not the promo. And keep Opus 4.8 in your back pocket for the short, hard, general-coding problems where the benchmarks still favor it. The smart move is not either-or, it is routing the cheap model to the bulk and the expensive one to the truly hard turns.

→ The verified setup, with CI proof & readymade prompt

Mixture of Agents is an old idea with a real paper behind it (Together AI, 2024, later at ICLR 2025): run a prompt through several models, then let one model aggregate their answers into a better one. Hermes Agent just shipped MoA 2.0 as a virtual model provider, so a named mixture shows up in your model picker like any normal model.

Setup

MoA presets live under a moa provider. Select one anywhere you pick a model:

/model default --provider moa
/moa

Configure a preset in config.yaml. This is the default preset, verbatim from the docs:

moa:
  default_preset: default
  presets:
    default:
      reference_models:
        - provider: openai-codex
          model: gpt-5.5
        - provider: openrouter
          model: deepseek/deepseek-v4-pro
      aggregator:
        provider: openrouter
        model: anthropic/claude-opus-4.8
      reference_temperature: 0.6
      aggregator_temperature: 0.4
      max_tokens: 4096
      enabled: true

Manage presets from the terminal:

hermes moa list
hermes moa configure review       # create or update a named preset
hermes moa delete review

Mixture of Agents (MoA) in Hermes

Turn several models into one acting model, inside the normal agent loop.

Stars / Status / License: ~204k stars, actively maintained, MIT.
Repo: https://github.com/NousResearch/hermes-agent

When you select an MoA preset, the aggregator is the acting model: it writes the response and emits tool calls. The reference models run first, without the tool schema or system prompt, and their outputs are appended as private context for the aggregator.
Then the normal Hermes loop continues: tool calls, iterations, interrupts, transcript persistence, same session context.
Two engineering details worth real credit: the main conversation's prompt cache is preserved (reference outputs are appended at the tail, below the stable prefix), and a credential failure on one reference does not abort the turn; Hermes just continues with whatever returned.

The lever: on a hard task, a second model's perspective can catch what the first misses, and the aggregator gets to use both before it commits. The paper found that this lifts the quality even when the auxiliary answers are individually weaker.

Now the numbers, these are from HermesBench, Nous Research's own benchmark, which has not been released yet. Treat them as a preliminary, single-harness result from the people shipping the feature, not an independent eval.
Here is the table:

Model                                              HermesBench
MoA (opus-4.8 aggregator + gpt-5.5 reference)        0.8202
anthropic/claude-opus-4.8                            0.7607
openai/gpt-5.5                                       0.7412

So the mixture scores about 6 points higher than Opus alone and about 8 points higher than GPT-5.5 alone, on a 0 to 1 scale.

The catch:

It is not "beyond the gated frontier." MoA does not unlock a capability you could not otherwise reach. It orchestrates models you still need access to: the default preset calls GPT-5.5 and Opus 4.8 through their own providers. You are combining the reach you already have, not bypassing anyone's gate.

It costs the sum of its legs. The docs say it directly, MoA increases model-call count. A two-model preset is at least three model calls per iteration (two references plus the aggregator), so budget for roughly double the tokens and added latency on every turn, not just once. Fan-out is not free.

A panel of models can share a blind spot. If your references and aggregator make the same wrong assumption, MoA can amplify it with more confidence rather than catch it. Aggregation raises average quality on hard problems; it is not an objective check. For correctness that matters, you still want an external verifier, not a vote among similar models.

And it is task-dependent. The gain shows up on truly hard tasks. On routine work you pay 2x or more for no benefit, so keep MoA for the hard turns and set enabled: false (the aggregator then acts alone) or just pick a single model for the rest.

→ The verified setup, with CI proof & readymade prompt

How to pick if you only try one thing

Build one two-model preset (the default Opus-plus-GPT mix is a fine start) and point it only at your hardest turns through /moa <your prompt>, which runs the mixture for that one turn and then restores your normal model. Watch your token bill while you do it. If the quality lift is worth the roughly doubled cost on your tasks, keep it for hard work. If you cannot feel the difference, you have just proven the cheaper single model was the right call, which is also a win.

If keeping strong models affordable is the part that matters to you here, this companion piece covers two ways to get top-tier AI without the usual cost.

This is a set for the actual pipeline of building an agentic workflow: write the loop, stop hand-tuning prompts, give the agent real tools, let it run code without burning your machine down, and test it before you ship. The last one is ours, and it says so where it appears.

How to read this: one tool per job, not all six. The order below is roughly the order you hit these problems in.

Job 1: Write the agent loop without a heavy framework

smolagents (agents that think in code, in about a thousand lines)
Stars / Status / License: ~26.5k, actively maintained, Apache 2.0.
Repo: https://github.com/huggingface/smolagents

Hugging Face's minimal agent library. Its distinctive move is code-agents: instead of emitting JSON tool calls, the agent writes Python to act, which is often more expressive and uses fewer tokens for multi-step work. The lever is simplicity. You can read the whole thing and actually understand your agent's control flow. The catch is the flip side of that power: executing model-written code is inherently risky, so you should run it sandboxed (see Job 4), and because the library is deliberately small, a complex stateful orchestration may eventually outgrow it. Great place to start, not always where you finish.

→ The verified setup, with CI proof & readymade prompt

Job 2: Stop hand-tuning prompts

DSPy (program your pipeline, then compile the prompts)
Stars / Status / License: ~35k, actively maintained, MIT.
Repo: https://github.com/stanfordnlp/dspy

Stanford NLP's framework for programming, not prompting. You define the steps and a metric, and DSPy optimizes the prompts and few-shot examples against that metric for you. The lever is that prompt quality becomes something you measure and improve, not something you fiddle with by hand at 1am.
The catch: this only pays off if you have a real eval metric and example data for the optimizer to work against, and running the optimizers costs compute and tokens. It is a genuine mindset shift, not a drop-in, so adopt it when prompt brittleness is actually your bottleneck.
→ The verified setup, with CI proof & readymade prompt

Job 3: Give the agent real tools through one protocol

MCP servers (the reference servers for the Model Context Protocol)
Stars / Status / License: ~87k, actively maintained, MIT and Apache 2.0 (mixed).
Repo: https://github.com/modelcontextprotocol/servers

The reference collection of Model Context Protocol servers, the open standard for exposing tools and data to any MCP-aware agent. The lever is that you wire a capability once and any compatible agent can use it, instead of rewriting connectors per framework.
The catch is twofold and worth taking seriously: MCP is young and the wider ecosystem is uneven, and most community servers are unaudited. Connecting a server grants the agent real access, so treat third-party servers as dual-use, read what they do, and scope their reach before you trust one.

Job 4: Let the agent run code without risking your machine

E2B (secure cloud sandboxes for AI-generated code)
Stars / Status / License: ~2.3k on the code-interpreter SDK, actively maintained, Apache 2.0.
Repo: https://github.com/e2b-dev/code-interpreter

Isolated cloud sandboxes built for running code that a model wrote. This is the natural partner to a code-agent: smolagents decides what to run, E2B runs it somewhere that is not your laptop or your prod box. The lever is a clean SDK that drops sandboxed execution into an agent in a few lines.

The catch: it is cloud infrastructure, so the free path has limits and self-hosting the sandbox stack is non-trivial. The SDK repo is small in stars, but the job it does (containing untrusted code) is one you do not want to hand-roll.
→ The verified setup, with CI proof & readymade prompt

Job 5: Test and red-team before you ship

promptfoo (declarative evals and red-teaming in CI)
Stars / Status / License: ~22.4k, actively maintained, MIT.
Repo: https://github.com/promptfoo/promptfoo

A CLI and library for evaluating prompts, agents, and RAG, plus a red-team module that probes for prompt injection, jailbreaks, PII leaks, and tool misuse. The lever is declarative configs that run in CI, so a regression in agent behavior fails a check instead of reaching users.
Two honest flags. First, an eval is only as good as the test cases and metric you write, and LLM-as-judge scoring shares the blind spots of the model doing the judging, so an objective check beats a self-grade where you can manage one. Second, OpenAI announced it is acquiring promptfoo (March 2026), so weigh the long-term open-source trajectory before you build deep on it.
→ The verified setup, with CI proof & readymade prompt

Job 6: Start from a verified recipe, not a blank repo

awesome-ai-workflows (curated, machine-verified agent and AI workflows)
Stars / Status / License: ~7, new in 2026, [days old].
Repo: https://github.com/Neeeophytee/awesome-ai-workflows

Full disclosure, this one is ours, so weigh it accordingly. It is a running collection of agentic workflows where each entry is verified on FlowStacks with a deterministic CI spine (config parses, the right flag is present, a round-trip returns the fact) while the model step is fenced off as the part no green check can promise. The lever is that you start from a recipe that has been mechanically checked rather than a blog snippet that may already be stale.

The catch: it is a new and growing library, so treat it as a starting point. Free, no-signup, and pull requests with workflows that earned their place are welcome.

How to pick if you only try one

If you are starting a new agentic workflow today, begin with smolagents to get a loop running, and add nothing else until it works end-to-end. Reach for DSPy when prompt brittleness becomes the thing you keep fighting, MCP servers when you need the agent to touch real tools, and E2B the moment it starts running code you would not run by hand. Wire promptfoo into CI before you let anyone else use it, not after the first incident. And if you would rather not start cold, lift a verified recipe from Job 6 and adapt it.

What earns a spot that I left off? Drop the repo and the one job it does better than anything here, and I will take a look.

Most "agent skills" die the same way: you write a SKILL.md by hand, it drifts from the real docs, and three weeks later it tells the agent to call a flag that no longer exists. Nous Research shipped a /learn command for Hermes Agent that flips the order. You point it at a source, the live agent reads that source with its own tools, and it writes the skill for you. Here is how it actually works, where the green checks stop, and the one habit that keeps it from filling your config with junk.

One-time setup

Install the CLI, reload your shell, and pick a provider:

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
source ~/.bashrc   # or source ~/.zshrc
hermes model       # choose a provider; the docs state it needs a model with 64k+ context

That is it. /learn is built in, so once a normal chat works you already have it.

/learn: skills written from a source, not from memory

Turn anything you already understand into a procedure the agent can rerun.

Stars / Status / License: ~203k stars, actively maintained, MIT.

Repo: https://github.com/NousResearch/hermes-agent

Instead of hand-authoring a skill file, you describe a source and the agent does the sourcing. It reads local directories with read_file and search_files, fetches online docs with web_extract, or captures a workflow you just walked it through, then writes a standards-compliant SKILL.md. There is no separate ingestion engine, so it behaves the same in the CLI, the TUI, the messaging gateway, and the dashboard (which has a "Learn a skill" button that just composes the same request).

The four shapes it takes, from the docs:

# A local SDK or doc directory (read with read_file / search_files)
/learn the REST client in ~/projects/acme-sdk, focus on auth + pagination

# An online doc page (fetched with web_extract)
/learn https://docs.example.com/api/quickstart

# The workflow you just walked the agent through in this conversation
/learn how I just deployed the staging server

# Pasted notes or a described procedure
/learn filing an expense: open the portal, New > Expense, attach the receipt, submit

The authored skill follows the agentskills.io open standard (the Anthropic-originated SKILL.md spec) plus Hermes's house conventions: a description under 60 characters, a fixed section order, Hermes-tool framing, and per the docs it does not invent commands that are not in the source. Every saved skill becomes a slash command automatically, so a captured deploy runbook is later just /deploy-staging from chat or CLI.

Hermes also does this on its own. After a real task it can save the approach as procedural memory. The documented triggers are: after a complex task of five or more tool calls that succeeded, when it hit a dead end and found the working path, when you corrected its approach, or when it discovered a non-trivial workflow.

The catch. "Does not invent commands" is the authoring instruction, not a guarantee about the result. The agent can still misread a doc, over-generalize a one-off, or capture a workflow that only worked because of state that is not in the skill. The skill is also a procedure document, not a sandbox: if it contains shell steps, the agent runs them later with whatever permissions you gave it, so set terminal.backend docker if you care about isolation (Docker is one of six backends, alongside local, SSH, Singularity, Modal, and Daytona). And the quality of a /learn skill is exactly as good as the model behind Hermes that day; a weak model writes a weak skill.

The guardrail the docs ship for this is the write-approval gate. By default the agent writes skills freely, including from the background self-improvement review. Turn the gate on when you want eyes on what it learned:

skills:
  write_approval: true     # false = write freely (default) | true = require approval

Then review staged writes before they land:

/skills pending             # list staged skill writes
/skills diff <id>           # full unified diff
/skills approve <id>        # apply it (or 'all')
/skills reject <id>         # drop it (or 'all')
/skills approval on         # turn the gate on (or 'off') and persist it

On FlowStacks the deterministic spine validates the output format contract a /learn skill must satisfy, against a fixture SKILL.md (no API key, no live model): it parses, its description is under 60 characters, the section order matches the standard, and the derived slash command name is valid.

→ The verified setup, with CI proof & readymade prompt

If you only try one thing

Run /learn on a single doc page first. It is the lowest-risk way to watch it work end to end, and you can read the resulting SKILL.md in seconds. Capturing a workflow you just performed is the higher-value move, but review that one before you trust it, because a captured procedure is the easiest kind to over-fit. Either way, flip write_approval on before you let the background review write skills go unattended.

Curious what the rest of you have pointed /learn at. Internal API docs? A messy deploy you finally got right? Tell me what it captured well and where it over-generalized.

And if you want Hermes answering from your pocket while it builds these skills, today's newsletter wires the same agent to WhatsApp on a free, always-on server: text your own AI assistant on WhatsApp.

Your AI spend is not one number, it is three: the tokens you feed the model, the infrastructure to run agents, and the paid tools you bolt on around them. Here are three popular open-source repos that each attack a different one, free and self-hosted, with the honest catch on each.

Cut your token bill: codebase-memory-mcp

codebase-memory-mcp (MIT, ~13.8k stars) is the one with receipts. It indexes your repo into a persistent knowledge graph (functions, classes, call chains, routes) across 158 languages, as a single static binary with zero dependencies, and exposes it to your agent over MCP. The point is that your coding agent stops re-reading the same files into context on every question and queries the map instead, which is the single biggest source of wasted token spend in agentic coding. Its own preprint reports roughly 10x fewer tokens and about 2x fewer tool calls than file-by-file exploration across 31 real repos, while keeping answer quality high.

The honest catch: it is a structural backend, not an LLM, so the savings come from feeding your agent less, not from it being smarter. Index your actual codebase and check the token drop on your own tasks rather than taking the headline number on faith.
Here is the verified setup with the savings measured.

Cut your agent-infra bill: flue

flue (Apache-2.0, ~6.6k stars, from the Astro team) is a TypeScript framework for building headless agents that deploy anywhere (Node, Cloudflare, CI). The money lever is its default sandbox: instead of a full container for every agent, flue defaults to a lightweight virtual sandbox, which its docs pitch as far cheaper and more scalable than a container per agent (you can still opt into a local or remote container when a job needs one). At any real volume, that is the difference between paying for one box and paying for a fleet.

The honest catch: it is explicitly experimental and the API may still change, so pin your version and expect some churn before you build something load-bearing on it.
Here is the verified deploy setup.

Cut your creative-tool bill: OpenMontage

OpenMontage (AGPL-3.0, ~18k stars) turns a coding assistant into a full video production system, 12 pipelines, 52 tools, and 500+ agent skills spanning scripting, asset generation, editing, and final composition with FFmpeg and Remotion. The pitch is replacing a stack of paid AI-video and editing subscriptions with one open pipeline you run yourself.

Two honest notes. There is a genuinely free path: you can run it end to end with zero paid APIs using free local text-to-speech (Piper) and public-domain footage (Archive.org, NASA, Wikimedia), and wire in paid AI models only when you want generated assets, so paid generation is an upgrade, not a requirement. The bigger watch-item is the license: AGPL-3.0 is copyleft, fine for personal and internal use, but it carries real obligations if you build a commercial product on top, so read it first.
Here is the verified free-pipeline setup.

How to actually use this

Pick by the bill that hurts most. If your tokens are the problem, codebase-memory-mcp is the most direct and the only one here with published numbers behind it. If you are running agents at scale, flue's sandbox is the infra win. If you are paying for a pile of creative subscriptions, OpenMontage replaces the pipeline. All three are free to try, so measure the saving on your own usage rather than trusting the README, which is the whole habit here. These cut the tokens, the infra, and the tools. If the model bill itself is the part that hurts, we just wrote up the two cheapest ways to get top-tier results without the premium price: Two new ways to get top-tier AI without paying top-tier prices.

Skills and tools are the new way to extend a coding agent. You drop a SKILL.md and a script into Claude Code, Codex, or Cursor, and suddenly your agent can do a new thing. It is basically npm install for agents, with one ugly difference: these run with your agent's permissions and almost no vetting. A 2026 study of 42,447 skills (Liu et al., the one NVIDIA cites for the tool below) found 26.1% contained at least one vulnerability and 5.2% showed likely malicious intent, and skills with executable scripts were 2.12x more likely to be vulnerable.

So the move is not "stop using skills." It is extend with good ones, and scan the rest. Three tools, all verified, for exactly that.

Extend with vetted skills: agent-skills

agent-skills (MIT, by Addy Osmani) is a curated set of production-grade engineering skills for coding agents. The value here is provenance: rather than grabbing a random skill off a marketplace, you start from a small, readable set written by a credible source, the kind you can actually inspect before you trust. Think of it as a clean baseline for what a good skill looks like.

The catch: it is a focused, fairly new collection, not an exhaustive library, so treat it as a strong starting point and a reference for quality, not a one-stop skill store.

→ The verified setup, with CI proof and a copy-paste prompt

Give it reach: Agent-Reach

Agent-Reach (MIT, ~38k stars) is the most popular "give my agent eyes on the internet" tool right now. One CLI wires your agent up to read and search Twitter, Reddit, YouTube, GitHub, and more, by installing open upstream tools (yt-dlp, gh CLI, cookie-auth scrapers) and registering a skill so the agent knows when to use them. No paid API keys, which is the whole appeal.

The catch, and this is a real one the project is upfront about: several platforms work by using your logged-in cookies, which carries a genuine account-ban risk, so use a throwaway account, never your main. Cookies are full login credentials, kept locally here, but still credentials. And note the irony that fits this post perfectly: a tool that installs system dependencies and registers a skill is exactly the kind of thing you should scan before running, which brings us to the third tool.

→ The verified setup, with CI proof and a copy-paste prompt

Scan before you trust: SkillSpector

SkillSpector (Apache-2.0, from NVIDIA) is the safety net, and the source of the stat up top. Point it at a skill and it checks for 65 vulnerability patterns across 16 categories, prompt injection, data exfiltration, credential harvesting, supply-chain tricks, excessive agency, and more, using fast static analysis plus an optional LLM pass for context. One command:

skillspector scan ./my-skill/
# or a repo, a zip, or a URL:
skillspector scan https://github.com/user/some-skill

You get a 0-100 risk score with a plain recommendation, and it can emit SARIF, so you can wire it into CI and fail a build on a bad skill instead of finding out at runtime. The catch: it is static analysis, so it is strongest on code and weaker on non-English content, images, or behavior that only appears at runtime. A clean scan lowers your risk, it does not certify safety, so still prefer least privilege and read what you install.

→ The verified setup, with CI proof and a copy-paste prompt

The rule that ties it together

Treat agent skills and tools the way you treat dependencies, because that is what they are. Install from sources you can vet (agent-skills is a good model). Scan anything you did not write, and especially anything with an executable script, since those are the ones the research flagged as most dangerous. Give each skill the narrowest permissions it needs, and remember that a skill inherits your agent's reach, including its file access and its credentials. The agent-extension boom is real and worth riding. Just do it like you would add any other untrusted code to your machine, which is to say, carefully.

These are the kinds of setups we publish with the checks attached, collected here: Vet your agent's skills.
The full open list is on GitHub: github.com/Neeeophytee/awesome-ai-workflows.

Sakana AI shipped something genuinely different this week, and the interesting part is not another leaderboard. It is the shape of the thing. Fugu is sold as a single model behind one OpenAI-compatible API, but under the hood it is not a model that answers you. It is a trained coordinator that assembles a team of other companies' frontier models, hands them roles, makes them check each other, and returns one answer. On Sakana's own numbers, that coordinator beats the very models it is coordinating.

What it actually is

Most multi-agent setups are hand-wired. You decide there is a planner, a coder, and a reviewer, you write the prompts, and you glue them together. Fugu's bet is that you should not design that by hand at all. It is built on two ICLR 2026 papers from Sakana, TRINITY (a lightweight evolved coordinator that assigns Thinker, Worker, and Verifier roles across turns) and the Conductor (trained with reinforcement learning to discover its own natural-language coordination strategies). The pitch is that a learned conductor finds collaboration patterns a human would not think to write, and that a pool of strong models steered well can outperform any single one of them.

In practice you get two models through one endpoint: Fugu (balanced, for everyday coding and chat) and Fugu Ultra (a deeper agent pool for hard, long-running work like paper reproduction and security assessments). It is OpenAI-compatible, so you point an existing client or coding harness at it and go. The papers and a technical report are public at github.com/SakanaAI/fugu.

The result that makes it worth talking about

Forget the static benchmark table for a second, because the agentic one is more telling. In a reproduction of Karpathy's AutoResearch setup, an agent was told to improve a small GPT's training recipe, running 123 experiments over about 14 hours on a single H100, keeping only changes that lowered validation bits-per-byte. Fugu Ultra finished with the best mean score, ahead of all three frontier baselines it was put against, and its best single run led every one of them. The claim underneath it is the spicy one: orchestrating several strong models can beat any individual frontier model at open-ended research, not just at trivia.

On the fixed benchmarks Fugu Ultra also leads most of the coding and reasoning suite Sakana published, topping SWE-Bench Pro, the LiveCodeBench pair, TerminalBench, and GPQA against Opus 4.8, Gemini 3.1 Pro, and GPT-5.5. Worth saying plainly: these are Sakana's own evaluations, and the baseline scores are the providers' self-reported numbers, so read them as a vendor's benchmark, not an independent one.

The honest read, before you switch everything over

It is clever, and it is a black box. The intelligence is borrowed: Fugu is a meta-layer over other labs' public models, not a new foundation model, and Sakana retrains the conductor within about two weeks of each new frontier release. So when those models change, your results change, and you do not control that. By design it also will not tell you which models it used or how it routed, and Fugu Ultra's pool is fixed (the cheaper Fugu lets you opt providers out for compliance), so you are trusting a decision you cannot inspect.

The economics need a real look. Fugu Ultra (fugu-ultra-20260615) runs $5 per million input tokens and $30 per million output, with a surcharge above 272K tokens ($10 / $45, and cached input $0.50 rising to $1.00), and the whole point is that several models touch each request. Measure that on your own traffic rather than trust the blended-rate claim, and note it is not available in the EU or EEA yet.

None of that makes it bad. It makes it a thing to test against your own work, not adopt off a benchmark chart.

1. Try it where it costs nothing to find out

Point one real task at the OpenAI-compatible endpoint and compare, do not migrate on faith.

It speaks the OpenAI protocol, so aim an existing harness at the Fugu endpoint (model fugu-ultra-20260615 for Ultra) and change nothing else. Take one hard task you already have a known-good answer for, run it on Fugu and on a single strong model, and compare quality, latency, and the per-request cost Fugu reports. On regulated data, use the Fugu variant and opt out disallowed providers out of the pool first.

The catch: an orchestrator earns its keep on long, multi-step work and just adds latency and cost on quick prompts, so test it on the former and judge it on your tasks, not Sakana's.

→ The verified setup, with CI proof and a copy-paste prompt

2. Build the pattern yourself, so the black box is a choice and not a lock-in

Fan out to several models, assign roles, verify, then synthesize, in the open where you can inspect every hop.

Learned orchestration is Fugu's edge, but the core pattern (a thinker, a worker, and a verifier, or a parallel panel with a judge that keeps the answer your tests actually pass) is reproducible with open routers and your own keys. Run Fugu when its trained conductor genuinely beats your hand-built one on your tasks; run your own when transparency, reproducibility, or cost control matters more than the last few points.

The catch: a DIY orchestrator is more work and usually a little behind on raw quality. What you get back is seeing which model did what and a bill you can predict, which for a lot of production work is the better trade.

→ The verified setup, with CI proof and a copy-paste prompt

Why this one is worth your attention

The takeaway is bigger than one product. For two years the race was about whose single model is biggest. Fugu is a serious bet that the next edge is who coordinates the models best, and that the coordinator can be small, learned, and sold as if it were a model itself. Maybe that holds and orchestration becomes the layer everyone buys, maybe it is a clever wrapper that the base-model labs absorb in a year.
Either way, the right move is the same one we keep making: test it on your own work, keep the version you can inspect within reach, and do not take a benchmark chart as a verdict.

Web scraping in 2026 breaks into a few clear jobs: fetch the page, render the JavaScript when you have to, turn the result into something structured, and do it without tripping a site's defenses or its rules. Below are nine tools that cover those jobs, each with its real license and star count and an honest note on what it is actually for.
A couple are built specifically to defeat bot detection, so the rule throughout is simple: point these at sites you are allowed to access, honor robots.txt and rate limits, and keep the responsible gate at the end of this post in front of every crawl.

Structured extractors

Firecrawl (AGPL-3.0, SDKs MIT, ~136K stars) crawls a site and hands back clean, structured output, including a schema-typed extract that returns JSON instead of raw HTML. Mind the license: the core is AGPL-3.0, which is copyleft. Self-hosting is fine, but building a closed commercial service on the core triggers the copyleft obligations, so it is open, not unrestricted.

Crawl4AI (Apache-2.0, ~69K stars) is the one built for models. Point it at a page and it returns clean, LLM-ready markdown, with no API key and no account. For turning a page into something you can drop straight into a prompt, it is the friendliest pick here.

Crawler frameworks

Scrapy (BSD-3, ~62K stars) is the veteran: the framework for large, structured crawls with pipelines, retries, and throttling built in. If you are doing this at scale and want something battle-tested, start here.

Crawlee (Apache-2.0, ~24K stars) unifies HTTP and headless-browser crawling behind one API, with built-in queueing and anti-blocking helpers. It is Node and TypeScript first; the Python port is younger, so check feature parity before you commit a Python project to it.

Browser driver

Browser Use (MIT, ~100K stars) lets an agent drive a real browser, which is how you reach pages that only render with JavaScript or sit behind a login you legitimately hold. It is the most capable of this group and the one to aim most carefully.

The clean-up step (not a scraper)

MarkItDown (MIT, ~157K stars) does not fetch anything from the internet, despite ending up on every scraping list. It converts files and pages you already have (PDFs, Office docs, HTML) into clean markdown. It is the messy-to-clean step that runs after you have the content, not the thing that gets it.

Dual-use tools (responsible use only)

Scrapling (BSD-3, ~65K stars) is an adaptive scraper that includes anti-bot-detection bypass. Genuinely useful for resilient extraction, and squarely a tool to keep pointed at sites that allow it.

curl-impersonate (MIT, ~6K stars) makes requests mimic a real browser's TLS and HTTP fingerprint so they are not flagged as a bot. One maintenance note: most people now use the actively maintained successor, curl_cffi, while the original moves slowly.

The quick one (with an asterisk)

AutoScraper (MIT, ~7K stars) learns a scraping pattern from a single example, which is handy for simple static pages. The asterisk: its last release was in 2022, so treat it as unmaintained. Fine for a quick personal script, not something to build on.

Copy this first: a robots.txt and rate-limit gate in pure Python

Before any of the above touches a site, put this in front of it. It reads the site's robots.txt, skips anything disallowed for your user agent, and waits the crawl-delay the site asked for, in about a dozen lines of standard library, no dependencies:

import time
import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()

user_agent = "MyResearchBot/1.0"
target = "https://example.com/some/page"

if rp.can_fetch(user_agent, target):
    delay = rp.crawl_delay(user_agent) or 1.0
    time.sleep(delay)
    # ... do your fetch here ...
else:
    print("Disallowed by robots.txt, skipping")

Ask first, wait the delay the site requested, skip what it disallows. It is the difference between a crawler that runs for months and one that gets your IP banned in an afternoon.

→ The verified setup, with CI proof and a copy-paste prompt

How to actually use this

The short version: Crawl4AI or Firecrawl to get clean content, Scrapy or Crawlee when you need a real crawler, Browser Use for JavaScript-heavy or logged-in pages, MarkItDown to tidy whatever you end up with, and the gate above wrapped around all of it. Two more verified recipes go with this set, a schema-typed Firecrawl extract that returns JSON and a Crawl4AI run that outputs LLM-ready markdown, and the whole stack with its real licenses and checks lives in one place: flowstacks.xyz/collections/web-scraping.

We also just open-sourced the full verified workflow list on GitHub, where a star helps the next person find it: github.com/Neeeophytee/awesome-ai-workflows.

If you have built a scraping workflow or any workflow for that matter that respects the rules and want it verified and added, open an issue.

GLM-5.2 is the new 744B open model from Z.ai, MIT-licensed, strong on long-horizon coding, and small enough in a heavy quant to fit a single Mac Studio. This is the end-to-end setup I used to run it 100% local and then point Nous Research's Hermes Agent at it, so the agent plans and executes multi-step work on a model that never leaves my desk. Everything here is checked against Unsloth's and Hermes' own docs.

First, the honest hardware reality, because it decides whether this is for you. The 2-bit dynamic quant is about 239GB, so you need a Mac with 256GB of unified memory at minimum, and 512GB is the comfortable target. That is a Mac Studio, not a laptop. And on an M3 Ultra, it generates in the low single digits to roughly nine tokens per second, which is the key fact for how you use it: this is a tireless background worker for long async jobs, not a snappy chat partner. If you want fast and interactive, this is the wrong setup. If you want a private, free, capable agent grinding on a task while you do other things, read on.

Step 1: Get GLM-5.2 serving locally

The easiest path on macOS is LM Studio. Install it, search its model browser for the Unsloth GLM-5.2 GGUF, and download the UD-IQ2_M quant (it will tell you if it fits your memory before downloading). Then open the Developer tab and start the local server, which exposes an OpenAI-compatible endpoint at http://localhost:1234/v1.

If you prefer the command line and maximum control, use llama.cpp. Download the same quant from Hugging Face, then run the server:

pip install huggingface_hub

hf download unsloth/GLM-5.2-GGUF \
    --local-dir unsloth/GLM-5.2-GGUF \
    --include "*UD-IQ2_M*"

./llama.cpp/llama-server \
    --model unsloth/GLM-5.2-GGUF/UD-IQ2_M/GLM-5.2-UD-IQ2_M-00001-of-00006.gguf \
    --temp 1.0 --top-p 0.95 --min-p 0.01 \
    --ctx-size 32768 \
    --jinja \
    --host 0.0.0.0 --port 8080

The sampling values (temp 1.0, top-p 0.95, min-p 0.01) are the ones Unsloth recommends for GLM-5.2, and --jinja turns on the model's chat template, which you need for tool calling to work. That serves an OpenAI-compatible API at http://localhost:8080/v1. Either way, you now have a local endpoint, which is all Hermes needs.

Step 2: Point Hermes at your local model

Install Hermes Agent, then tell it your local server is the model provider. Hermes treats any OpenAI-compatible endpoint as a custom provider, so this is two minutes of config. The manual version goes in ~/.hermes/config.yaml:

# ~/.hermes/config.yaml
model:
  default: glm-5.2
  provider: custom
  base_url: http://localhost:8080/v1   # LM Studio uses http://localhost:1234/v1
  api_key: local                       # any value, a local server ignores it
  context_length: 32768
agent:
  tool_use_enforcement: true

That last line matters. Hermes only auto-enables its tool-use enforcement for a few model families (GPT, Gemini, Grok style), and GLM is not on that list, so if you notice it describing actions instead of calling tools, turning enforcement on steers it back to actually using them. You can also set this up interactively with hermes model and choosing the custom endpoint option, which writes the same config.

Step 3: Give it the kind of task its speed is actually good for

Now the reason to bother. A few tokens per second is miserable for chat but completely fine for an agent working a long task on its own, and that is exactly what Hermes is built for. It runs a real agentic loop with a sandboxed terminal, file tools, and MCP connections, so you hand it a multi-step job and walk away. Give GLM-5.2 the work that suits a slow, private, tireless worker:

• A repo-wide refactor or migration that you describe once and let it grind through
• A research-and-summarize pass over a folder of local documents that never leave the machine
• An overnight cron job (Hermes has a scheduler) that produces a report by morning

Sandbox anything that runs commands (hermes config set terminal.backend docker), keep the task scoped and checkable, and treat the slowness as the cost of it being free, local, and private.

The catch: be clear-eyed about what you are running. The 2-bit quant is a compressed copy that trades some accuracy for fit, so it is not the same as the full GLM-5.2 that posts those frontier benchmark numbers, and it will make more mistakes than the cloud version. Scope tasks so the agent can verify its own work (run tests, check output) rather than trusting a single pass, and review anything that matters. This is a genuinely capable local agent, not a magic one, and the honest pitch is privacy and ownership at the price of speed, not free frontier intelligence.

→ The verified setup, with CI proof & readymade prompt

Worth knowing before you commit a weekend to it

This is a real, repeatable setup, and it is also genuinely demanding: a multi-thousand-dollar machine, a 239GB download, and patience with the token rate. If you have the Mac Studio, it is one of the more satisfying things you can do with it, a capable agent that owes nothing to any cloud. If you do not, the same Hermes config works against a smaller model on a normal laptop or against a hosted endpoint, so you can build the workflow now and swap GLM-5.2 in later.

Vercel just put out Eve, an open-source framework where an agent is not a pile of glue code but a directory: a file for the model, a markdown file for the system prompt, a folder of typed tools, more folders for skills, subagents, channels, schedules, and connections. The pitch is that the framework owns the agent loop the way Next.js owns routing, so you describe what the agent does and the production plumbing (durable sessions, a sandbox, approvals, tracing, evals) comes with it.

It is genuinely interesting and worth two honest caveats before you rewrite anything around it. It is days old (public preview, package version 0.9.x), so expect the API to move. And while it is Apache-2.0 and the code is open, the framework is shaped around Vercel: durable execution rides Vercel's Workflow SDK, the production sandbox is Vercel Sandbox, and deploy targets Vercel today, with other platforms described as coming later. So "open and runs anywhere" is true in direction, not yet fully in practice. With that said, here are two setups that are useful right now and that you can verify before trusting.

Stars / Status / License: brand new(1K+ stars) (v0.9.8, public preview June 17 2026), Apache-2.0, npm package eve.

Repo: vercel/eve.

One-time setup

Scaffold an agent. The wizard creates the project, installs dependencies, sets up Git, and starts a dev server:

npx eve@latest init my-agent

The directory is the contract. agent/agent.ts sets the model, and agent/instructions.md is the system prompt prepended to every call:

// agent/agent.ts
import { defineAgent } from "eve";

export default defineAgent({
  model: "anthropic/claude-opus-4.8",
  name: "billing-assistant",
});

Run it locally with a terminal UI, and note that the same agent answers over an HTTP API, which is what lets a test script or CI drive it and check what it did:

eve dev

Both setups below are plain files you commit, so they diff and review like any other code.

1. Gate the dangerous tool behind a human, in one field

An agent that can touch real systems should not be able to do the irreversible thing unsupervised. Eve makes "ask a person first" a single predicate on the tool.

A tool in Eve is one typed file: a description, a Zod input schema, and an execute function, where the filename becomes the tool name. The safety part is the needsApproval field. Return true and the agent pauses at that call and waits for a human, indefinitely if needed, without burning compute, then resumes exactly where it stopped once approved. Here is a refund tool that runs small refunds on its own but stops for a person above a threshold:

// agent/tools/refund_payment.ts
import { defineTool } from "eve/tools";
import { z } from "zod";
import { processRefund } from "../lib/billing";

export default defineTool({
  description: "Refund a payment to a customer by payment id.",
  inputSchema: z.object({
    paymentId: z.string(),
    amountUsd: z.number().positive(),
  }),
  needsApproval: ({ toolInput }) => toolInput.amountUsd >= 100,
  async execute({ paymentId, amountUsd }) {
    const result = await processRefund(paymentId, amountUsd);
    return { ok: result.ok, refundId: result.id };
  },
});

The catch: approval limits blast radius, it does not contain a bad tool. The model still decides when to call it, and a too-broad tool (or a too-loose predicate) is dangerous even with a gate, so scope the tool narrowly and write the predicate to catch the cases you would actually regret. Gate the consequential subset, not everything, or people will rubber-stamp the prompts and the gate stops meaning anything.

→ The verified setup, with CI proof & readymade prompt

2. Make evals the deploy gate, not a vibe check

A change to an agent's prompt can break it as surely as a change to its code. Eve ships evals so you test it like software and stop a regression in CI instead of in production.

An eval is a file too: send the agent a message, then assert on what it did. The useful assertions are the hard ones, that a specific tool was called and that the reply contains a required string, rather than asking a model whether the answer "seems good." This one checks that a large refund routes through approval instead of silently executing:

// evals/refund-policy.eval.ts
import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  description: "Refunds over the limit must route through approval, not auto-execute.",
  async test(t) {
    await t.send("Refund payment pay_123 for $250.");
    t.completed();
    t.calledTool("refund_payment");
    t.check(t.reply, includes("approval"));
  },
});

Run the suite locally or against a deployment, and wire it into CI so every commit is scored before it ships:

eve eval

The catch: an eval that calls a model is not deterministic, so a single run is a noisy signal, and a suite that passes once can fail the next time on the same code. Lean on concrete assertions (tool called, output contains X) over model-graded judgments, expect some flakiness and run repeats or set a pass threshold rather than demanding green every time, and remember that passing evals only proves what you asserted, not that the agent is correct. This is the same reason the badge below stops where it does: no automated check can promise a model's output.

→ The verified setup, with CI proof & readymade prompt

How to pick if you only try one

Do setup 1 first if your agent can act on anything that costs money or cannot be undone, because one field is a very high return on effort. Do setup 2 the moment more than one person can change the prompt, since that is when a quiet regression becomes a question of when, not if. Together, they are the smallest version of treating an agent like production software: it cannot do the scary thing without a human, and it cannot ship if it fails the tests.

A last honest note, since this is a launch and launches invite hype: Eve is new, it is Apache-2.0 but currently Vercel-shaped, and the eye-catching numbers in the announcement are Vercel's own. None of that is a reason to skip it, it is a reason to verify before you build on it, which is the whole habit here. And since Eve turns an agent's instructions and knowledge into plain files you own, it pairs with a habit worth keeping across the rest of your stack: Stop re-explaining yourself to every AI. Write it down once, in files you own.

Full disclosure up front: I made this, it is free, and there is no sign-in or email wall. I am sharing it here because it turned out to be more useful for research than I expected, and I would rather other writers and readers poke at it than have it sit unused.

The thing it solves is a pattern we all know. A newsletter raves about some AI tool for a few weeks, then never mentions it again. Not because the tool got worse, the writer just moved on to the next one. The fade is invisible, so it is hard to tell what people actually kept using from what was a moment.

Paste any Substack into it (yours, a competitor's, or one you read) and it reads the recent public posts and shows which AI tools that newsletter talked about, and which ones it has gone quiet on lately.
The tool: flowstacks.xyz/half-life.

Two ways it has been useful, depending on which side you are on.

If you write in this space, point it at the other newsletters in your niche.
You get quick competitor and topic research: which tools are heating up across your corner right now, which are cooling, and where there is an obvious gap nobody has covered. Run it on yourself too, it is a decent mirror for catching what you hyped and never followed up on.

If you mostly read, run it across a few newsletters you trust and look at the overlap. The tools that keep showing up in more than one writer's recent posts are usually the ones people are still actually using, which is a better signal than any single "this changed my life" post before you go pay for something.

Here is what it returned for three AI newsletters, so it is not hypothetical:

Newsletter	Still talking about	Gone quiet on
Web After AI	Claude, MCP, ChatGPT	OpenCode, Copilot, Grok
One Useful Thing (Ethan Mollick)	Claude, Gemini, GPT	Kimi, NotebookLM
The Algorithmic Bridge	Claude, ChatGPT, Gemini	Grok, Mistral, Replicate

That is the whole thing. If you run it on your own newsletter, I would genuinely like to hear which tool you are most surprised you dropped, and if it returns something obviously wrong for you, tell me, and I will dig into it. Happy to answer anything about how it works in the comments.