u/Lower-Impression-121

• Evals = Tests.
• Harness = A Program.
• Production Grade = I couldn't make it work so I don't believe You.
• Cosine = A little Big Word to sound Big.
• Token = Sieze the means of Production!
• Hallucation = I didn't communicate my intent to the agent
• AI Slop = See Hallucination.
• Clanker = Roger Roger.
• RAG = Hello World.
• Guardrails = See Hallucination.
• Approvals = Never done Agent or integration at scale.
• Insecure = See Hallucination.
• Vibe/Prompt/Loop/Pray/Love Engineering = Not my Tribe.

Please add more.

Up Front! is a card based WW2 game by Avalon Hill, "the squad leader card game" released in the 80s. It has a very 80s rulebook. Everything is numbered. It has a hierarchy, a lot of cross-referencing, exceptions and a technical writing style. I was asked to post a howidid here.

It also has an active user base and a lot of rules query threads on the interweb. It is a good, well known, small rule corpus to build and verify a Rules Lawyer from. This was/is the process:

• Have a large body of authoritive Q&A/rulings or similar available. The less you have to make up the better. Discussion boards are good for this because there are multiple participants and the answer settled/agreed upon is likely the right one. It is Authoritive. Single response souces could be wrong or subjective.
• Turn the board(s) into Test Cases. I sicked the AI at boardgamegeek to and it wrote a program to scrape all the Rules questions. It produced a large json file with the original question, the 'this is the question I will ask', all the replies, and the consensus answer that the Rules Laywer RAG will be verified against.
• The RAG system was built, it had already gone through iterations to match the document structures - the nature of the domain - which imho is critical to understand. There's no point throwing a chunker at a document without knowing how the document Works. ie: How someone reads it. As I play these games I'm familiar with how to use them, which is why they make good learning prototypes that move to the gatekeeped 'production code' fairly easily.
  • It has Chroma, a schema/metadata input file (the RAG is to be generic), an index, a hierarchy TOC that the ingestion filled out. A Co-occurance graph was added after test runs to boost the score.
• As I use small models because of an old slow machine, and remember the days of building scripted websites in dailup (you could not get away with fat websites back then), I like to follow the philophosy of build to lower common demoninator and worst conditions: not the biggest machine or best LLM. Those hide performance, and in the case of LLMs, reliability problems if the user base isn't using the same size.
• Run a batch of tests (mainly because of speed. 90-95% of the query time is the LLM).
• Review the score, what was answered well, could be better, failed. Ask the AI - what can I do here? What are some recommendations? Did you account for X? (I knew the document structure better than the AI builder I had it put in cross references and exclusion clauses). Have that conversation, generate a plan and rerun. Small batches are good here too to limit the number of issues or improvements. Faster turn around to see if score went up or down.
  • Are the changes going to cause a reingestion or the corpus or retreival only? The further getting into it it was additions to ingestion that did not require full redo, and more on the retrieval side - effeciently finding all the sections and subsections.
  • It introduced HyDE (Hypothetical Document Embeddings) which made me recall days of grammer parsing. How many different ways can we say the same thing in english? The question layer has to take those variations into account to hit the vectors. It's like dialects, coffee vs kawfy. When youre dealing with freeform and rambling questions its inclusion was a big boost.
  • A preprocessor LLM that stripped out the actual question long questions ignoring preamble, filler, colour etc. Smaller than the reasoning model its goal is to produce "this is what the real question is" and in a format closer to how the rules are writen. Changing kawfy to coffee.
• Rebuild and rerun. When happy, move on to the next batch
• The score fluctuates. Rule, down, fix rerun up, preferrably higher than where it was. Big gains happened first, then its about keeping it there. Because the questions are random, a batch will have variety which reduces the risk to regression, although that's nothing more than spinning up another process if the resources are available.
  • current run went from 98 to 94 with a false negative thrown in, still iterative improvement time.
• The model doens't matter all that much. Weak models work, they're just slower and not as refined or depending on who made it, verbose. The answer will be right if the retrieval passed in the right data meaning the document structure was understood and well ingested to make retrieval find the right data.

tldr at the end

• find as large and authoritive a dataset to test against as possible before making it up.
• understand how to read the data that will ingested.
  • for example, glossy magazines split acticles all the time, have side text boxes, images; easily visually related, requires knowing to graph).
• test and refine in batches.
• deliberately use underpowered models to harden pipelines.
  • also test on better ones

finally.

a week or two ago there was a thread on single episodes and i commented 'game show': to which i have written into my longer headfic over chapters 11-13. its the most 'normal' cartoony jem v misfits where as the rest is heavier identity drama. for mature readers and for those willing to let characters be.... as the story requires.

Below is hypothetical, thought process.

Let's say this is the scenario: the NOrks (Waaaaaghh) compromise a Weather API and put a poison response into some small % of responses.

We call that weather api for some purpose. who cares what. we store the response in a DB. we give the response to Agent(s) who do their thing.

Getting a poison would cause something Bad to happen.

The DB is also used in BI, analytics... more agenty stuff, and plenty of users connect to where its data ends up as well, like Excel and Word and Copilot...

As the subject line says: where is the Taste Tester? where do you or should you put It?

myself and others who have a firm grasp of the future and have accepted it (if that wasnt inflamatory enough) poo-poo the peer review.

why? because we, you, they didn't write it and no one is going to read it except another bot. and by then its too late. even in the SDLC a good team didn't wait for a PR, they talked and got opinions continuously. really, changing a PR is like a production bug, costly. there was even a thing called extreme and peer programming.

no, let it go. just let it go. elsa was right.

review the Plan instead. that thing you tell your swarm of gremlins to produce before they do any meaningful work. review it Hard. make it review itself. get another llm if there are the tokens to review it. get it to explain itself, like you were sitting next to a peer in a Peer Programming session and have that conversation. What about this, have you covered that? can you think of any edge cases.

what you'd do when going over the PO or BA's requirements - you know those roles that will become extict because you will be doing them as well - grooming and refining. brush the hair, smooth out the tangles. give He-Man his MOTU mane.

there was another pre-cumbrian term for this: Shift Left. that is, move every possible cause of failure as close as to the requirements or ideation phase as possible and solve or get rid of there.

thats what you do in with the Plan. youre shift-lefting, your reveiwing the plan and grooming and exercising your own Bigger Picture mind to know what to tell the Team to look out and come back with a solution for. because if you can't tell them what to do well, they're not going to do it well and good luck fixing that up in the PR stage.

https://preview.redd.it/agzavlaihggh1.png?width=1637&format=png&auto=webp&s=df3c4df7baf17c6b66df5d1c28bbfd713b34df3f

what we're trying to do here is the whole memory-game; remembering useful actions, decisions and Things. Weighted (time based, adjustble), graphed and conversational/queryable along with tool access through the chat (tied to the user's proxy's permission set).

as i see it the objective of these things is to help people remember, connect dots and offer some level of advice. BE pipelines ingest in the user and their proxy's activity stream (mail,chat,process actions, whatever can be wired in and known to the user).

top of mind is entity explosion - far too many records to display in lists or the graph and whether the user can kind of make sense of it anyway.

tool calling - chat and things happen like an API called to get or change data, is what makes these things usefull, otherwisee theyre fairly limited to what they hold. assistants/secretaries have agency to do things. what is easy - but also not a replcaement for real UX because forms and APIs get large, validated, have rules etc - is dynamic form generation from the tool spec (say an MCP card).

here, great, dynamic, therefore can do for anything; user has to know those 5 values. we may need a conversation (with mara) to fill them out, which could require other Tool calls. another browser tab for a different conversation, a sub-conversation, a second conversation, how best to 'fill it in' without a C&P?

https://preview.redd.it/yoci7a1vbigh1.png?width=405&format=png&auto=webp&s=938f98968dfa805c8f5d01897932705c7856b6ba

this was the streaming link. unsure if it'll work when its done.

When the Gov shut down a model and said "national security" the world where we are going towards is the Past. While the majority of us are going: it can't code or how to RAG without falling over, mils and govs are going "How to Break Into Your System" and "How To Stop It". And as the techniques become known so will competitors use. Just as we see top talent being stolen (what was the plot for Count Zero?) across the Tier 1s we will see systems attack each other.

Why?

Because it will be Easy.

Maybe cheap. Like drone warfare in ukraine.

And it'll be geopolitical. Which happens Now.

Our legacy security will be outsmarted. We have had major data breaches in the past years. Without accepting the game has become F1 fast, or believing that code quality still matters, you will be leaving your company open for more and virulent cyber attacks.

Fighting it is adopting it. Encasing systems in fortified ICE which can observe and react faster than we can think, just as fast as the ICE-breakers, from script kiddies to open sourced nork utilities busters, try probe, breach, take or crash.

To "learn" RAG I (actually the IDE) under my (mostly its) instruction built a very powerful (according to the IDE) RAG system for very complex (they are, wargames rules are very complex and fought over - heaps of test cases for the eval uh, integration test suite) and technical documents.

It worked.

And got me thinking what to build next beyond the AI version of Hello World (which RAG is). Other than document or data analysis, what else is frameworkable, repeatable and should be a public repo instead of some startup idea because every org is going to have to do it and have one.

Just as the light bulb was wargames rules to RAG, the Game itself was a slap to the face. I have the Rules, I have the Game state - uh - Sim-u-lati-on (ppap reference there in case you missed it).

The Goal: To, using an underpowered laptop, no GPU, small Open Weight models, and a subscription plan to Antigravity (the self-harm is amazing), build (actually the IDE) a generic Simulation platform based on the rules in the aformentioned Rules RAG and custom game state (ie the map, units involved etc.) and have it optimise a turn, and play play adversarily against each other (against other Red Team Simulator).

It will be called Prime Radiant.

The Rules Set (Renegade Legion, which layers from Solar System Campaign to an RPG) is loaded.

And the end of it, I believe that if successful, the engine, which will also be shoved up onto github like the RAG, can be used for business simulation purposes such as the impact on sales if a product price was reduced, or what a competitors actions could be in response etc and many many other scenarios. (Its up to you, Dear Reader, to then provide your own Rules and "Game" state); and prove that OSS and LLMs can replace massively expensive proprietary systems.

After a conversation on whether Advanced Squad Leader (all versions) or Star Fleet Battles (plus Captains Log) was the better or harder end game, I have settled up Up Front! small enough, same rules format, some supplements and errata to start with and then progress ASL to SFB.

The objective is to have it answer accurately across versions and conflicts and overrides or special situations. If it can do that then we have an answer to the Rules Lawyer.

The activity will take a few days. Finding all the material will be the hardest part - boardgamegeek and wargamevault are first points of call. Assistance here would be greatly appreciated, I do this for the architecture not the content. My office wall of bookcase games attest.

After a conversation on whether Advanced Squad Leader (all versions) or Star Fleet Battles (plus Captains Log) was the better or harder end game, I have settled up Up Front! small enough, same rules format, some supplements and errata to start with and then progress ASL to SFB.

The objective is to have it answer accurately across versions and conflicts and overrides or special situations. If it can do that then we have an answer to the Rules Lawyer.

The activity will take a few days. Finding all the material will be the hardest part - boardgamegeek and wargamevault are first points of call. Assistance here would be greatly appreciated, I do this for the architecture not the content. My office wall of bookcase games attest.

https://preview.redd.it/aldj6vztcy8h1.jpg?width=1024&format=pjpg&auto=webp&s=7502333be3fa48fda7d432bdc9321700bae38a8c

Image and top-right Target State thesis expanded by gemini, collaborated with and refined by grok and chatgpt.

Quadrant Breakdown

• Top Right (Autonomous Enterprise Systems): Platforms that bridge high-reasoning autonomy with rigorous, process-centric governance. These systems prioritize organizational integrity, audit trails, and "Agentic JIT" execution over simple task-completion.
• Top Left (Autonomous Agent Frameworks): Code-first environments providing maximum flexibility for building complex agentic loops and state machines. They offer high reasoning power but require significant manual effort to graft on enterprise-grade security and process management.
• Bottom Right (Managed Work Platforms): Ecosystems that provide deep enterprise integration and stability. They excel in structured, deterministic automation but are historically rigid and limited in handling non-deterministic "reasoning" at scale. 
• Bottom Left (Workflow Automation): Platforms built for visual, deterministic integration. They are the standard for low-complexity, rule-based piping, serving as the connective tissue for data rather than the "brain" of the operation.

Detailed Tool Comparison

1. AWS AgentCore & Google Vertex AI (Autonomous Enterprise Platforms)

• Focus: Infrastructure-native governance and enterprise-scale orchestration.
• Strengths: Unmatched integration with cloud security, IAM policies, and VPC-level isolation. They leverage sophisticated "Agent-to-Agent" (A2A) protocols and Model Armor, providing the foundation for highly secure, multi-agent operations.
• Weaknesses: They remain "How-focused"—optimizing agent execution rather than the high-level business process. Achieving "Organizational Operational Correctness" still requires custom architecting above these layers.

2. LangGraph & CrewAI (Autonomous Developer Frameworks)

• Focus: Granular control over execution logic and state management through code.
• Strengths: Allows developers to craft intricate, multi-agent collaboration patterns that proprietary platforms often hide. Ideal for building custom AI logic that needs to be portable and highly modular.
• Weaknesses: They are raw engines, not turn-key platforms. They lack native support for enterprise audit logging, formal business process management (BPM), and pre-built Human-in-the-Loop (HITL) compliance dashboards.

3. Microsoft Azure AI Foundry / Copilot (Managed Work Platforms)

• Focus: Providing an enterprise-ready environment for AI that feels familiar to current organizational workflows.
• Current State: Deeply embedded in the M365 and GitHub ecosystem, moving from assistive "copilot" behavior toward more autonomous, background-task orchestration.
• Expected Trajectory: Aggressively moving to fill the gap between individual productivity and enterprise process control by leveraging their deep-seated footprint in existing corporate backends.

4. Claude Team / Cowork (Autonomous Enterprise Systems)

• Focus: Leveraging superior reasoning and "computer use" capabilities to handle complex, unstructured workflows.
• Current State: Leads in pure agentic reasoning and context analysis. It operates as a highly intelligent team member, though it currently sits "on top" of business systems rather than fully integrating into the legacy, process-heavy backends of a traditional enterprise.
• Expected Trajectory: Evolving from a "smart workspace" into a formal orchestrator by expanding its API surface area and integrating more tightly with enterprise protocol standards.

Summary

The market is converging. While frameworks like LangGraph and CrewAI provide the core logic engines, the "Top Right" is being contested by two different strategies: Cloud Providers (AWS/Google) are building outward from security and infrastructure, while Agentic Innovators (Claude/Expert-Proxy.com) are building inward from reasoning and process-centricity.

me: hmmm. are song lyrics too.... hidden meaning driven to be useful as prompts?
gemini: I cannot fulfill this request. (x2)

back context: i was mentioning how listening to darkwave helped (moved over from lofi) the thought processes. gemini was going on about how the genre meta matched the architetural style (as these things do to make you feel good about yourself) which lead to a short back and forth about communication quality being the role nowadays between Fingers and Circuits, and as music was on the mind I asked the above and it couldn't handle it.

Maybe AI's are zentraedi.