u/siddharthnibjiya

Hello everyone, sharing some of the terms that businesses & Agentic developers in my network think about while building Agents / agentic systems. Might help some of you to box your existing agent development problems and discuss them more easily!

1. Human Steering: Does your agent get from start to end without real-time or blocking human prompting? And I mean start to end, you must truly think if it's getting from A --> B and then human takes the process from B -> C and then agent does C --> D OR agent helps go from A--> D. Your goal needs to be to strategically think if there's value and leverage in removing steering in your use-case, vis-a-vis the risk/probability of agent-error > human-error.

2. Context Poisoning: When irrelevant or unintended context leaks into the agent's working set and pulls it off task: a random instruction in a document which was for another scenario, a stale process or poorly matched embedding in a vector search.

3. Agentic Search: When you don't pre-inject context/RAG into the agent but actually give the agent tools to self-discover it's knowledge base and identify the right information. [Caveat: here the assumption is that you have the right information easily discoverable by the agent. You can't dump it all files and tell it to search - I mean you can, but it's not going to reap the impact that it typically can when done right]

4. Non-determinism: When your agent behaves or responds differently for exact same prompt in a similar environment/situation.

5. Domain Verifiability: Whether the response given by your agent can be deterministically identified as correct or incorrect, or if there's nuance/subjectivity. Also, sometimes if it's deterministic but takes a human the same or similar time as it would to actually do it manually, then the verifiability is moot.

6. Behavioral Evals: When a domain is unverifiable or correctness is not measurable, you looking at whether the agent followed the right process, did the right thing, went down the right reasoning path, etc.. - like eval the process, not just the outcome.

7. Context Erosion: If the agent is solving problems in an environment that's changing continuously, overtime, an agentic system that was giving x% accurate answers can drop to x-delta% accurate answers without even any change in the system.

8. Provenance: Can the final response given by the agent be grounded in facts or cited to sources as assumptions, helping create a good trace for the agent.

9. Autonomy levels: Is your agent a Waymo or is it Tesla FSD? There's like nuanced levels and autonomy completely changes on the scale at which you're looking at things too. But the point is, can you define the job for which the agent was built and truly yourself evaluate what level you're at? I had made a matrix in the past for an agent I built and I knew it was close to L4 than to L0 but was like L2.5 at that point.

Thanks!

https://preview.redd.it/u11ekzys0nih1.png?width=1080&format=png&auto=webp&s=580d6318e710d7f56b313d6f8d5f1d075d5c50c3

I've been a heavy openclaw user for 8 months now.. Sharing my story here.

I am an engineer but mainly look at business now in my startup - sharing how I scaled GTM since I setup OpenClaw in January.

I do it myself --> I get one of my openclaw agents to do most of the heavylifting in that work (examples below) --> when I lose the bandwidth of managing the agents, I start hiring folks for a role who would then end up managing those goals with the agents as underlying way of doing work.

Work that my agents do pretty well:

1. Events & conference planning / scanning / prioritising
2. Lead & funnel management - active conversations // followups // todos
3. Investor mgmt
4. Mailing list updation & mgmt
5. Managing hiring pipeline

Almost every work (read and write) related to g sheets, gdocs. and every analysis/read work related to CRM, Calendar, sales.

What I still don't get agents to do:

1. Define intent or write copy. Anywhere - the alpha in defining your goals and strategy is unparalleled - especially if you have understanding of doing smth.
2. Send emails or calendars or do explicit communication to humans.

What really has helped me rolling here: Adding additional access to the agent as and when needed:
- gdrive
- calendar
- mail
- Sybill (sales call transcripts)
- browser with logged in linkedin
- Brave search api (this one is an absolute beast unlock - reduced 95% of browser usage i think)!!!

moved from openai and claude models to kimi.

Any questions?

I've been a heavy openclaw user for 8 months now.. Sharing my story here.

I am an engineer but mainly look at business now in my startup - sharing how I scaled GTM since I setup OpenClaw in January.

I do it myself --> I get one of my openclaw agents to do most of the heavylifting in that work (examples below) --> when I lose the bandwidth of managing the agents, I start hiring folks for a role who would then end up managing those goals with the agents as underlying way of doing work.

Work that my agents do pretty well:

1. Events & conference planning / scanning / prioritising
2. Lead & funnel management - active conversations // followups // todos
3. Investor mgmt
4. Mailing list updation & mgmt
5. Managing hiring pipeline

Almost every work (read and write) related to g sheets, gdocs. and every analysis/read work related to CRM, Calendar, sales.

What I still don't get agents to do:

1. Define intent or write copy. Anywhere - the alpha in defining your goals and strategy is unparalleled - especially if you have understanding of doing smth.
2. Send emails or calendars or do explicit communication to humans.

What really has helped me rolling here: Adding additional access to the agent as and when needed:
- gdrive
- calendar
- mail
- Sybill (sales call transcripts)
- browser with logged in linkedin
- Brave search api (this one is an absolute beast unlock - reduced 95% of browser usage i think)!!!

moved from openai and claude models to kimi.

https://preview.redd.it/inf2gsi70nih1.png?width=1346&format=png&auto=webp&s=65c4072ae637315c9bb8776a070f68f3bc6797c5

1. Human Steering - In a prompt that requires agent to think and navigate a 10 step "decision tree" thinking process, how many steps can it navigate by itself with the context, without

2. Context Poisoning - Out of any 10 random scenarios, how often does your agent get lost between markdown files and instructions, ending up creating an incoherent/un-useful answer?

3. Determinism - Given the same scenario and prompt, and you run the same sequence 5-10 times, how many times does the agent come up with the same / similar answer?

4. Context Updation - When there is new information coming to the context, how is it appended into the exisint g knowledge base? And are you able to avoid conflicts in case the new context updates an existing fact?

5. Accuracy -.. the feedback you receive from users on the correctness of the response (if the field is verifiable ofcourse, otherwise the feedback is more vibes)

What else do you all use to measure context?

Anyone here who only uses vibes ?

Please stop using RAG for critical / complex use-cases 🙏 where domain knowledge needs to be navigated or searched!

I was building a complex AI Agent, focused on production alert RCAs.. The context in which agent operated was in a knowledge base of 200+ skill documents, all of which had interlinkages like Obsidian.

It was really bad in responses without human guidance because the agent would start with one document and end up in some other documents very soon - the facts and ideas it started getting was varying quite a bit.

We did a small pivot in the approach.. We did a quick data modeling of our use-case.. In our case, the most important entities are services, dashboards, infrasturcutre_components, repositories, alerts, runbooks.

We took that data --> put into an opensearch index.

Then after that, lets say when an alert came, instead of automatically identifying the best related context and injecting into the agent, we did the other way around.

WE GAVE THE AGENT A TOOL TO SEARCH THE CONTEXT.. The tool had 5 parameters (query, filter, search_type, temporal, limit)..

This combination of (a) Saving context in structured JSON document (b) making agent search it got the agent on a good track for improvement..

Did you ever face such issues with Markdown? How did you handle it?

tl;dr Don't inject custom context basis user query/RAG/etc. into prompt, make agent search it with a tool with params (query, filter, search_type, temporal, limit). It was the single biggest lever to bring control on using the agent..

I build AI agents at my company, and initially, our context layer was obsidian stlye skills markdown files folders, cross-links. We would initially do vector search / RAG and inject the context alongside prompts. We used to see repetitive challenges there and we went down rabbit hole trying to fix it.. What kept breaking:

• Context poisoning / digression. Once we were past ~50 markdown files, the agent would wander between docs and pick up instructions that had nothing to do with the task. We tried building explicit navigation paths and interlinking everything, but it didn't help much.
• No source proof. As the knowledge base grew, we couldn't reliably say which piece of context drove a given action. Users won't trust an agent that can't show its work.

What actually worked for us:

1. Structured docs instead of markdown. We moved context into JSON / structured documents. Agents navigate way better when things look like code. We had about 10-12 document types and then each type had 5-8 fields within them
2. Make the agent search, don't spoon-feed it. Instead of pre-injecting context, we gave it meta-info about what context exists and made it responsible for searching and discovering the right pieces (tool-based search capability for the agent rather than us running RAG/prompt expansion upstream).

For search, we created a tool search_resources that would run queries on the opensearch index in which the structured docs were stored - the tool we created had 5 parameters:

* query - Select the query it wants to run

* filter - Filter by specific type of documents

* search_type - Define search type (semantic / syntactic)

* temporal - Add temporal True/False if your data has time based staleness

* limit - number of responses it receives in return

If you're doing something similar, what's your experience been?

What else is working great for you?