u/moxie-docs

I benchmarked a tool I've been building across a fork of the Full Stack Example App - the results were pretty interesting to me, and revealed some ideas for future testing.

My hypothesis was that providing an MCP where agents get all of the conventions of a codebase directly - it would result in reduced token spend because they would "get it right" the first time and it would not require asking it to make changes, or revisions after a review of the code.

Some TL;DR on the findings:

• Token cost was more with Moxie Docs - this immediately surprised me and then I realized what was happening. The MCP we provide (and the AGENTS.md context) instructs agents to identify if their changes impact or warrant documentation updates. This predictably results in more token usage from the agent using the MCP versus control - because the agent is updating and adding documentation automatically.
• I need more complex test cases - I used some basic API endpoint updates, small features, etc. that were in hindsight relatively trivial for most agents nowadays, so the result was not a strong convention drift that can be seen with larger one-shot attempts / prompts, and it did not result in higher impacts to docs going stale
• Use real world repos - I have metrics from our own users (merge rate of PRs, comments tagged to us to update document generation, signals from our Q&A feature feedback, etc.) that are more impactful to look at, but I think the Full Stack Example app repo is a good holistic project scaffold, but not indicative of a real world app with stale docs, missing docs, etc.

Would love any feedback or to answer questions! I plan to re-run this experiment in the future with some improvements to the methodology but as a quick sanity check on impact it's very promising. I've also found the need to set up eval harness workflows on our models and prompts to detect drift on our quality - one thing that was interesting was bumping up the thinking level on some models actually resulted in worse output, something I did not expect.

For those SaaS builders that are using AI-tooling - do you document the code they output?

How do you do it? Are you asking the AI to document its own work, using a third-party tool, doing it yourself? Genuinely curious if people are writing their own docs to better understand what the code being shipped does, or if docs are ignored because it's maintained by AI?

I've personally found that having solid docs (and a good way to surface them to agents) improves consistency and quality, I'd say I'm at maybe 30% AI code 70% hand-written, but I do have 100% docs being generated.

I launched my own startup around 6 years ago (scheduling software for universities) and although it lasted quite a few years I really only got one customer, but it was an awesome experience to learn and grow a product like that. About a month ago I got really sick of using Claude Code and having it forget things all the time (like me telling it not to add useless comments) and I tried a few different memory systems but at the end of the day realized the biggest gap in the codebase I worked on was documentation.

What it does:

• Moxie Docs indexes your GitHub repo and pulls out your codebase conventions, documentation, missing docs / doc gaps, stale documents, etc.
• Gives you a fully fledged MCP hook-in for your favorite tools (Claude Code, Codex, Cursor, etc.) and opens a PR into your repo to update or add an AGENTS file with instructions
• The MCP gives your agents the ability to "get it right" the first time - by matching codebase conventions, automatically identifying & updating documents in the same PRs you work on, and continually improving your codebase quality.
• We summarize & gather PRs merged to create a very easy to use Changelog feature (you can choose date range, include items, view things shipped internally like dev tooling vs external user facing features, and copy changelogs to share to users easily)
• You can have Moxie Docs automatically rewrite all PR descriptions (from any user in the repo) to match a standardized format and highlight anything missing.
• Gives

I'm pretty happy that (at least I think so) I was able to make a product that would benefit both "traditional" engineers like myself building by hand, as well as heavily AI-assisted development flows by splitting documentation into AI-focused MCP tooling and user-friendly features for engineers.

Would love any feedback / input / questions, and if you are inclined to want to upvote / comment on PH I'd appreciate: https://www.producthunt.com/products/moxie-docs (just launched today)