u/ParsleyMaximum1702

Hey everyone,

I found myself wanting to move past the typical black-box framework implementations and map out the exact numerical and structural properties of attention mechanisms from scratch. I ended up building a full validation workbook tracking the evolution from early Seq2Seq models to the foundations of the modern Transformer.

I turned this into an open-source technical masterclass and printable workbook. If you are looking to cement your core mathematical understanding of attention layers, this deep dive steps through:

1. Tensor Dimensional Validation: Mapping coordinate frameworks to ensure valid tensor alignments before execution.
2. Track A (Additive / Bahdanau): Manual execution of linear projections into a common latent workspace, non-linear activation steps, and scalar reductions.
3. Track B (Multiplicative / Luong): Step-by-step calculation of direct matrix dot-product interactions using a bilinear bridge weight matrix.
4. The Softmax Normalization Ledger: A visual analysis of how different alignment formulas control focus distribution (soft blending vs. sharp gating).
5. The Vanishing Invariant Trap: Tracing the backward pass using the quotient rule to show how unscaled logits collapse the local gradient to zero, explaining the statistical and intuition-based necessity of the 1/sqrt(dk) scaling factor.
6. Parallel Hardware Realities: Breaking down how Luong attention unifies sequences into a single matrix product, bypassing sequential loops to maximize GPU efficiency.

I have included the full step-by-step master answer key in the text, alongside a link to download the blank PDF workbook if you want to print it out and do the tensor math by hand.

Check out the full article here:

https://open.substack.com/pub/ayushmansaini/p/tracing-the-math-of-seq2seq-attention?r=4zl69k&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Hey,

We’ve all seen the tutorials preaching the power of Worker-Critic multi-agent setups. But in production, without strict deterministic bounds, you hit a massive architectural wall: The Infinite Hallucination Trap.

If your agents are stuck optimizing for competing constraints, they can easily enter an endless reflection loop—burning tokens, inflating your context window, and running up insane API bills.

To understand exactly why this happens under the hood, I spent this weekend breaking down a dual-agent debugging loop entirely BY HAND using pencil, paper, and state error matrices. No LangChain, no framework fluff—just raw token mechanics.

Here is the breakdown of the first-principles tracing exercise I put together for Workbook 4 of my engineering series:

1. THE SCENARIO

We track an automated multi-agent patch system trying to fix a legacy multi-threaded bug under two conflicting constraints:

- Constraint A: Eliminate a memory leak (No dangling pointers)

- Constraint B: Maintain thread safety (No race conditions)

2. THE SYSTEM MATRIX DISCOVERY

- At t=1: The Worker generates Patch_v1. Leak resolved, but thread safety is broken (E_thread = 4).

- At t=2: The Critic catches the error. The Worker over-corrects with a heavy global mutex, shifting the stack allocation frame. Thread safety is fixed, but the leak is completely re-introduced (E_leak = 4).

- At t=3: The Worker panics, strips the mutex, rolls back to a version of Patch_v1, and the system resets back to the exact numerical state of t=1.

3. THE MATHEMATICAL TRAP

By tracking the progress delta (Delta E = |E_t - E_{t-2}|), we can mathematically prove when the system hits a dead stop. At step t=3, Delta E drops to an absolute 0.0, yet the overall system error remains stuck at E_t = 4.

The agentic system’s velocity collapses to zero before reaching a valid production state. It’s trapped in a perfect, non-converging limit cycle error orbit.

4. THE BARE-METAL CIRCUIT BREAKER

To solve this without throwing generic execution exceptions, I mapped out a deterministic Circuit Breaker Gate in raw Python that checks this exact zero-velocity threshold and freezes the system state matrix natively before the API call chain loops infinitely.

I’ve uploaded a full walkthrough article including the raw Python simulation code, a solved reference matrix, and an empty workbook PDF if you want to work through the token tracking math at your own lab bench.

I'd love to hear how you guys are natively catching non-convergence in your agent architectures!

👇 [Link to the Full Substack Breakdown & Free Workbook PDF in the Comments]

https://open.substack.com/pub/ayushmansaini/p/inside-the-infinite-hallucination?r=4zl69k&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Hey

I’ve been spending a lot of time breaking down modern LLM orchestrations down to bare-metal mechanics, inspired by the "AI by Hand" educational movement.

A common issue I see in enterprise multi-agent architectures (using LangGraph, CrewAI, etc.) is the tendency to naively append concurrent memory state data strings sequentially into the next prompt layer. This wastes massive token arrays, dilutes transformer attention allocation, and frequently triggers state hallucinations when identical semantic keys hold conflicting values.

To understand exactly how programmatic state synthesis impacts computational costs under real-world string noise, I created and traced a first-principles manual workbook to track the underlying variables.

I wanted to share the completed math trace and open-source the blank templates for anyone looking to drill down into the mechanics.

The System Profile Under Evaluation:

We simulate a text environment where two asynchronous nodes push conflicting values for identical state variables:

* Agent A (Detective Node): {"Joker_Location": "Arkham Asylum", "Threat_Level": "Low"}

* Agent B (Intelligence Node): {"Joker_Location": "Gotham Energy Plant", "Threat_Level": "Critical"}

What’s Covered in the First-Principles Trace:

1. Concurrency Fan-Out Topologies: Mapping out the parallel processing data flows and identifying the precise cross-contamination bottleneck area within a shared central engine graph.

2. Semantic Contamination Audit: Tracking token footprint inflation (127 characters for the naive stack vs. 69 characters for the single normalized schema schema).

3. Levenshtein Distance Matrix Integration: Tracing out a cell-by-cell dynamic programming matrix by hand to resolve input typos ("Arkhahm" vs "Gotham") and pinpointing the exact minimal alignment path (4 operations).

The Optimization Yield:

By computing direct structural state synthesis deterministically at the engine layer before runtime compilation, the payload context space is compressed by exactly 45.67%. Scaling this calculation out across enterprise production cycles directly correlates to slashed context costs and a significant drop in Time-To-First-Token (TTFT) latency.

Resources:

Because handwritten pencil grids can be tough to read on a mobile screen, I have structured the entire solved workbook into a clean, comprehensive markdown format in my article below, alongside a download link for the blank PDF practice sheets for your own practice files.

https://open.substack.com/pub/ayushmansaini/p/multi-agent-frameworks-are-bleeding?r=4zl69k&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

I would love to get your feedback on this architectural layout—how are you currently handling state arbitration and optimization in your concurrent multi-agent production loops?

Hey everyone,

I've been studying transformer prompt constraints from a first-principles approach, trying to move past just copy-pasting API endpoints and library wrappers.

To look at what actually happens when we merge parallel agent threads, I manually traced the token mechanics of a concurrent Map-Reduce pipeline (146 words total) on a scratchpad. I used a mock scenario where different agents track a crisis at Oscorp Tower and pass their messages back to an orchestrator.

The results really highlighted the reality of the "Lost in the Middle" phenomenon:

1.The agent that found a structural building collapse had the most critical update (Raw Score 9/10).

2. But because it got appended into the middle lane (position p=3), the transformer's position embeddings hammered it with a major attention decay penalty (alpha = 0.30).

3. Its final share of the attention mass collapsed down to just 11%—meaning it was mathematically drowned out by basic system instructions and formatting parameters.

I wrote up the full operational breakdown step-by-step showing exactly how to map out these prompt boundaries, compute raw-to-adjusted weight equations, and visually track the U-shape curve.

I also created a blank, printable PDF workbook layout so people can practice working out token contextshares on paper.

I'm trying to share more of this "AI by hand" style work. If you find this useful, you can check my Substack newsletter to get the printable workbook and join the community.

Link to the Substack is below! Let me know what you think of this methodology or if you’ve faced similar context challenges in production!

https://open.substack.com/pub/ayushmansaini/p/firing-ai-agents-in-parallel-made?r=4zl69k&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Hey everyone,

Modern agentic frameworks like LangChain or CrewAI make spinning up automated workflows incredibly easy, but their heavy abstraction layers often obscure the underlying algorithmic state transitions, memory overhead, and inference costs.

To understand exactly how sequential reasoning handles context payloads, I followed Andrej Karpathy's build-from-scratch ethos and Prof. Tom Yeh's "AI by Hand" approach. I completely mapped out a multi-hop ReAct (Reason + Act) trajectory onto a physical scratchpad workbook before writing any code. Then, I wrote a zero-dependency Python engine using pure standard libraries to programmatically verify the handwritten token counts and geometric matrices down to the fourth decimal place.

I wanted to share the structural mechanics and financial metrics of what happens under the hood during a simple 3-hop directed network query.

1. The Architecture & Topology Map

The scenario runs a search query over a simple 4-node directed knowledge graph layer to see if a hidden structural connection exists between characters:

[User Query] ───> [Prompt Context Window Buffer (S_t)] ───> [LLM Evaluation Loop]
                             ▲                                       │
                             │                                       ▼
                     [Tool Execution] ◄─── [Vector Match Registry] ◄─┘

Graph Topology: Batman -> Superman -> Iron Man -> Spider-Man

---

2. Manual Geometric Vector Routing

Instead of pulling in an external embedding database model, tool intent resolution is handled through manual 2D vector cosine similarity math against explicit tool coordinate profiles (Query vector vs. specialized tool profiles):

• Cosine Similarity Formula: Similarity(A, B) = (A · B) / (||A|| · ||B||) = (A1B1 + A2B2) / (√(A1²) + A2²) · √(B1² + B2²))
• Query State: q = [0.10, 0.90]
• Calculator Profile: t_0 = [0.95, 0.05]
• Graph Lookup Profile: t_1 = [0.15, 0.85]

Calculating the explicit dot products and scalar magnitudes yields an argmax selection value of 0.9980 for the Graph_Lookup tool versus 0.1625 for the calculator, triggering a clean tool execution route.

---

3. Visualizing Context Window Explosion

The core value of tracking memory arrays by hand is seeing the exact math behind context inflation. Because agents rely on an append-only state transition recurrence sequence, the prompt payload inflates rapidly with each iterative step:

• Memory Growth Rule: S_n = S_n-1 + T_n-1 + A_n-1 + O_n

Here is the exact step-by-step word count ledger from the workbook:

Timestep (t)	Structural Component Added	Step Words	Cumulative Payload Size (S_t)
0	Base System Prompt + Query (S_0)	22 words	22 words
1	Model Output Turn 1 (T_0 + A_0)	12 words	34 words
2	Environment Tool Observation (O_1)	5 words	39 words
3	Model Output Turn 2 (T_1 + A_1)	22 words	61 words
4	Environment Tool Observation (O_2)	6 words	67 words
5	Final Processing Sequence Block	49 words	116 words

---

4. System Diagnostics & Cost Modeling

To map how this context inflation hits financial budgets, I applied a standard tracking rate (C_in = $0.001/word, C_out = $0.003/word) using the explicit formula:

• Billing Formula: Cost_t = (Input Volume_t × C_in) + (Generated Volume_t × C_out)
• Context Explosion Ratio (rho): 5.27x expansion from initial query payload state.
• Turn 1 Expense (t=1): $0.0580
• Turn 2 Expense (t=3): $0.1050
• Turn 3 Expense (t=5): $0.2140
• Total Agentic Trajectory Cost: $0.3770

---

Why Build This?

Stepping away from frameworks and manually computing these tokens reveals the true cost and friction points of agentic loops. It shows why runtime costs scale quadratically or exponentially over long multi-hop paths if you aren't optimizing prompt cache states or tracking cumulative token growth turn-by-turn.

I have uploaded the full open-source verification framework, terminal logging scripts, matplotlib data visualization modules, and the high-resolution workbook worksheets to GitHub for anyone who wants to audit the math or fork the code.

Full Codebase and Worksheet Scans: https://github.com/Ayushman125/react-agent-from-first-principles

Hey everyone,

Modern agentic frameworks like LangChain or CrewAI make spinning up automated workflows incredibly easy, but their heavy abstraction layers often obscure the underlying algorithmic state transitions, memory overhead, and inference costs.

To understand exactly how sequential reasoning handles context payloads, I followed Andrej Karpathy's build-from-scratch ethos and Prof. Tom Yeh's "AI by Hand" approach. I completely mapped out a multi-hop ReAct (Reason + Act) trajectory onto a physical scratchpad workbook before writing any code. Then, I wrote a zero-dependency Python engine using pure standard libraries to programmatically verify the handwritten token counts and geometric matrices down to the fourth decimal place.

I wanted to share the structural mechanics and financial metrics of what happens under the hood during a simple 3-hop directed network query.

### 1. The Architecture & Topology Map

The scenario runs a search query over a simple 4-node directed knowledge graph layer to see if a hidden structural connection exists between characters:


[User Query] ───> [Prompt Context Window Buffer (S_t)] ───> [LLM Evaluation Loop]
                             ▲                                       │
                             │                                       ▼
                     [Tool Execution] ◄─── [Vector Match Registry] ◄─┘

Graph Topology: Batman -> Superman -> Iron Man -> Spider-Man

---

2. Manual Geometric Vector Routing

Instead of pulling in an external embedding database model, tool intent resolution is handled through manual 2D vector cosine similarity math against explicit tool coordinate profiles (Query vector vs. specialized tool profiles):

• Cosine Similarity Formula: Similarity(A, B) = (A · B) / (||A|| · ||B||) = (A1B1 + A2B2) / (√(A1²) + A2²) · √(B1² + B2²))
• Query State: q = [0.10, 0.90]
• Calculator Profile: t_0 = [0.95, 0.05]
• Graph Lookup Profile: t_1 = [0.15, 0.85]

Calculating the explicit dot products and scalar magnitudes yields an argmax selection value of 0.9980 for the Graph_Lookup tool versus 0.1625 for the calculator, triggering a clean tool execution route.

---

3. Visualizing Context Window Explosion

The core value of tracking memory arrays by hand is seeing the exact math behind context inflation. Because agents rely on an append-only state transition recurrence sequence, the prompt payload inflates rapidly with each iterative step:

• Memory Growth Rule: S_n = S_n-1 + T_n-1 + A_n-1 + O_n

Here is the exact step-by-step word count ledger from the workbook:

Timestep (t)	Structural Component Added	Step Words	Cumulative Payload Size (S_t)
0	Base System Prompt + Query (S_0)	22 words	22 words
1	Model Output Turn 1 (T_0 + A_0)	12 words	34 words
2	Environment Tool Observation (O_1)	5 words	39 words
3	Model Output Turn 2 (T_1 + A_1)	22 words	61 words
4	Environment Tool Observation (O_2)	6 words	67 words
5	Final Processing Sequence Block	49 words	116 words

---

4. System Diagnostics & Cost Modeling

To map how this context inflation hits financial budgets, I applied a standard tracking rate (C_in = $0.001/word, C_out = $0.003/word) using the explicit formula:

• Billing Formula: Cost_t = (Input Volume_t × C_in) + (Generated Volume_t × C_out)
• Context Explosion Ratio (rho): 5.27x expansion from initial query payload state.
• Turn 1 Expense (t=1): $0.0580
• Turn 2 Expense (t=3): $0.1050
• Turn 3 Expense (t=5): $0.2140
• Total Agentic Trajectory Cost: $0.3770

---

Why Build This?

Stepping away from frameworks and manually computing these tokens reveals the true cost and friction points of agentic loops. It shows why runtime costs scale quadratically or exponentially over long multi-hop paths if you aren't optimizing prompt cache states or tracking cumulative token growth turn-by-turn.

I have uploaded the full open-source verification framework, terminal logging scripts, matplotlib data visualization modules, and the high-resolution workbook worksheets to GitHub for anyone who wants to audit the math or fork the code.

Full Codebase and Worksheet Scans: https://github.com/Ayushman125/react-agent-from-first-principles

Hey everyone,

Modern agentic frameworks like LangChain or CrewAI make spinning up automated workflows incredibly easy, but their heavy abstraction layers often obscure the underlying algorithmic state transitions, memory overhead, and inference costs.

To understand exactly how sequential reasoning handles context payloads, I followed Andrej Karpathy's build-from-scratch ethos and Prof. Tom Yeh's "AI by Hand" approach. I completely mapped out a multi-hop ReAct (Reason + Act) trajectory onto a physical scratchpad workbook before writing any code. Then, I wrote a zero-dependency Python engine using pure standard libraries to programmatically verify the handwritten token counts and geometric matrices down to the fourth decimal place.

I wanted to share the structural mechanics and financial metrics of what happens under the hood during a simple 3-hop directed network query.

### 1. The Architecture & Topology Map

The scenario runs a search query over a simple 4-node directed knowledge graph layer to see if a hidden structural connection exists between characters:


[User Query] ───> [Prompt Context Window Buffer (S_t)] ───> [LLM Evaluation Loop]
                             ▲                                       │
                             │                                       ▼
                     [Tool Execution] ◄─── [Vector Match Registry] ◄─┘

Graph Topology: Batman -> Superman -> Iron Man -> Spider-Man

---

2. Manual Geometric Vector Routing

Instead of pulling in an external embedding database model, tool intent resolution is handled through manual 2D vector cosine similarity math against explicit tool coordinate profiles (Query vector vs. specialized tool profiles):

• Cosine Similarity Formula: Similarity(A, B) = (A · B) / (||A|| · ||B||) = (A1B1 + A2B2) / (√(A1²) + A2²) · √(B1² + B2²))
• Query State: q = [0.10, 0.90]
• Calculator Profile: t_0 = [0.95, 0.05]
• Graph Lookup Profile: t_1 = [0.15, 0.85]

Calculating the explicit dot products and scalar magnitudes yields an argmax selection value of 0.9980 for the Graph_Lookup tool versus 0.1625 for the calculator, triggering a clean tool execution route.

---

3. Visualizing Context Window Explosion

The core value of tracking memory arrays by hand is seeing the exact math behind context inflation. Because agents rely on an append-only state transition recurrence sequence, the prompt payload inflates rapidly with each iterative step:

• Memory Growth Rule: S_n = S_n-1 + T_n-1 + A_n-1 + O_n

Here is the exact step-by-step word count ledger from the workbook:

Timestep (t)	Structural Component Added	Step Words	Cumulative Payload Size (S_t)
0	Base System Prompt + Query (S_0)	22 words	22 words
1	Model Output Turn 1 (T_0 + A_0)	12 words	34 words
2	Environment Tool Observation (O_1)	5 words	39 words
3	Model Output Turn 2 (T_1 + A_1)	22 words	61 words
4	Environment Tool Observation (O_2)	6 words	67 words
5	Final Processing Sequence Block	49 words	116 words

---

4. System Diagnostics & Cost Modeling

To map how this context inflation hits financial budgets, I applied a standard tracking rate (C_in = $0.001/word, C_out = $0.003/word) using the explicit formula:

• Billing Formula: Cost_t = (Input Volume_t × C_in) + (Generated Volume_t × C_out)
• Context Explosion Ratio (rho): 5.27x expansion from initial query payload state.
• Turn 1 Expense (t=1): $0.0580
• Turn 2 Expense (t=3): $0.1050
• Turn 3 Expense (t=5): $0.2140
• Total Agentic Trajectory Cost: $0.3770

---

Why Build This?

Stepping away from frameworks and manually computing these tokens reveals the true cost and friction points of agentic loops. It shows why runtime costs scale quadratically or exponentially over long multi-hop paths if you aren't optimizing prompt cache states or tracking cumulative token growth turn-by-turn.

I have uploaded the full open-source verification framework, terminal logging scripts, matplotlib data visualization modules, and the high-resolution workbook worksheets to GitHub for anyone who wants to audit the math or fork the code.

Full Codebase and Worksheet Scans: https://github.com/Ayushman125/react-agent-from-first-principles