u/rangerrrr

A problem I suspect a lot of you have hit: agent memory stores remember whatever the model decides to remember, hallucinations included. Once a made-up "fact" is in durable memory it gets retrieved and repeated for weeks.

Confidence scores don't save you. A mis-calibrated model will hand you a high-confidence claim that simply isn't in the source.

What we do: a claim never becomes durable memory on confidence alone. It has to pass a source-grounding check that's model-independent. At least one of the claim's cited evidence snippets has to actually be present in the source text it was extracted from (normalized substring match, with a token-overlap fallback for punctuation/format drift). If nothing the model cited is in the source, the claim stays a reviewable "signal" and never auto-promotes.

It's deliberately lenient. A false negative is just an extra human review; we never want a false positive (an auto-promoted hallucination). High-impact claim types (forecast, commitment, deal risk) also need independent corroboration or a human even when grounded.

The honest limitation we're still tuning: lexical grounding proves the quote exists, not that the inference is correct. A real quote can back a wrong inference. So it's a floor, not the whole trust model.

Disclosure: this is from an open-source project I work on (CRMy), Postgres-backed.

Curious how others here gate what gets written to long-term memory. Anyone doing semantic/NLI-based grounding instead of lexical, and is it worth the latency?

I've been building agents for sales/CS workflows and kept hitting the same wall: the demo is great, but nobody will actually point it at a real customer w/o keeping a human in the loop (HITL).

What finally clicked is that it's not a context problem, it's a trust problem. Inside any account, the "context" you feed the agent is a mix of what the customer actually said, what someone inferred, what was true last quarter, and what the model made up. To a retriever those all look identical, so the agent treats a hallucination like a signed commitment.

The one that pushed me over the edge: an agent congratulated a customer on "expanding with the platform" based on a real note from a deal that had churned two quarters earlier. The note was real. It just wasn't true anymore.

What actually helped was treating customer knowledge the way a good rep does, with four things the raw context doesn't carry:

• provenance (did the customer say this, or did we infer it?)
• freshness (a champion or a next step has a shelf life)
• action boundaries (drafting is fine; sending or writing to the CRM needs a check)
• proof (what did the agent rely on, and what changed)

Disclosure: I work on an open-source project (CRMy) that does this, so I'm biased.

More interested in how the rest of you handle it: are you keeping agents off stale/made-up customer data in the prompt, in retrieval, or as a separate layer?

Disclosure: I’m the maintainer of CRMy. It’s open source. This is not a paid product pitch; I’m trying to get feedback from people using Claude Code in real workflows.

I built CRMy because I kept running into a gap between “Claude Code can do the task” and “Claude Code has the operational context to do the task reliably.”

For coding, Claude can inspect the repo, read files, run tests, and make changes.

But for customer/revenue workflows, the equivalent context is scattered everywhere:

• CRM records
• call notes
• emails
• stale research
• open follow-ups
• prior agent runs
• human approvals
• account history
• deal stage changes

So the agent either asks the same questions repeatedly, rebuilds context from scratch, or writes updates without enough state.

CRMy is my attempt at giving Claude Code a typed customer memory layer through MCP.

After setup, Claude Code can call tools for:

• contact/company/opportunity context
• full customer briefings
• activity logging
• assignments and handoffs
• human approval requests
• stale context review
• scoped writes
• audit history

Example setup:

claude mcp add crmy -- npx u/crmy/cli mcp

The main primitive is briefing_get.

Instead of asking Claude to query five different things before it acts, it can ask for a briefing on a contact, company, opportunity, or use case and get the relevant record, recent activity, open assignments, context entries, stale warnings, and related objects.

The design goal is not “CRM chatbot.”

It’s more like: if Claude Code is going to act as an operator, it needs a durable state layer that is structured enough to trust, but still usable through natural language.

Why I built it:

I think agents need typed memory, not just bigger context windows.
I think human handoff and approval should be first-class, not an afterthought.
I think customer data should be self-hostable and inspectable.
I think MCP tools need better operational state behind them, not just more APIs.

Where I’d love Claude Code-specific feedback:

1. Would you actually use Claude Code for customer/revenue workflows, or keep it strictly for dev work?
2. Does this kind of MCP server help, or does it just add tool clutter?
3. Should there be fewer high-level tools instead of many granular CRM tools?
4. What would make you trust an agent to write customer data?
5. What guardrails would you expect before using this with real contacts/deals?
6. Would examples inside CLAUDE.md / skills matter more than the tool API itself?

GitHub: https://github.com/crmy-ai/crmy
Docs: https://crmy.ai/

I’m especially interested in negative feedback from people who have tried MCP servers for sales or customer use cases and found them annoying, brittle, or too noisy.