u/dmpiergiacomo

Quick question for anyone running evals on their agents: when you optimize, are you tuning the parts (prompts, context blocks, retrieval, individual tools, etc.) or the whole (the full harness: logic + context together)?

My hunch is most teams start with the parts because it's tractable, but the real wins are at the whole-system level, where the parts interact and a local optimum isn't a global one. Curious whether that matches your experience or not.

If you're optimizing the whole harness: how do you actually do it? Which evals do you use, if any? Would love to hear your playbook. And if any of it is open source, please drop a link. Always more useful to learn from real examples.

Quick question for anyone running evals on their agents: when you optimize, are you tuning the parts (prompts, context blocks, retrieval, individual tools, etc.) or the whole (the full harness: logic + context together)?

My hunch is most teams start with the parts because it's tractable, but the real wins are at the whole-system level, where the parts interact and a local optimum isn't a global one. Curious whether that matches your experience or not.

If you're optimizing the whole harness: how do you actually do it? Which evals do you use, if any? Would love to hear your playbook. And if any of it is open source, please drop a link. Always more useful to learn from real examples.

Asking to the ones who care about evals.

What's the main thing you're trying to evaluate and optimize right now?

• Prompts?
• Context?
• The harness itself?

Most people I talk to still point evals at individual prompts. But my read is that the frontier has moved: the interesting work now is closing the loop and optimizing the entire context and/or harness, not tuning prompts in isolation.

Anyone already doing this in practice? Curious what your setup looks like and where it breaks down.

Hey folks who care about evals (as everyone should!!!)

What's the main thing you're trying to evaluate and optimize right now?

• Prompts?
• Context?
• The harness itself?

Most people I talk to still point evals at individual prompts. But my read is that the frontier has moved: the interesting work now is closing the loop and optimizing the entire context and/or harness, not tuning prompts in isolation.

Anyone already doing this in practice? Curious what your setup looks like and where it breaks down.

TL;DR: If you put many examples in one prompt and they share a block of text (a system prompt, instructions, a schema), don't copy-paste it into every example. Instead, write it once and reference it. In my tests it's free on simple tasks and measurably better on a harder "match each example to its own data" task, especially as the batch grows and on weaker models.

---

The two ways to render the same prompt

Three examples that share one system prompt.

Inline — the shared block is copy-pasted into every example (notice it appears 3×):

<example index="1">
<turn role="system">You are a helpful weather assistant. Be concise and accurate.</turn>
<turn role="user">What's the weather in Rome?</turn>
<turn role="assistant">18°C, light rain.</turn>
</example>
<example index="2">
<turn role="system">You are a helpful weather assistant. Be concise and accurate.</turn>
<turn role="user">What's the weather in Tokyo?</turn>
<turn role="assistant">31°C, sunny.</turn>
</example>
<example index="3">
<turn role="system">You are a helpful weather assistant. Be concise and accurate.</turn>
<turn role="user">What's the weather in Oslo?</turn>
<turn role="assistant">4°C, snow.</turn>
</example>

Reference — written once, pointed to (id="sys" declares it, var="sys" points to it):

<shared id="sys">You are a helpful weather assistant. Be concise and accurate.</shared>


<example index="1">
<turn role="system" var="sys"/>
<turn role="user">What's the weather in Rome?</turn>
<turn role="assistant">18°C, light rain.</turn>
</example>
<example index="2">
<turn role="system" var="sys"/>
<turn role="user">What's the weather in Tokyo?</turn>
<turn role="assistant">31°C, sunny.</turn>
</example>
<example index="3">
<turn role="system" var="sys"/>
<turn role="user">What's the weather in Oslo?</turn>
<turn role="assistant">4°C, snow.</turn>
</example>

Same information either way. With 3 short examples it barely matters — but scale to 50–100 examples with a real system prompt and the inline version balloons, and (the surprising part) the model starts losing track of which example lines up with which data.

---

Where I hit this

I'm building a context-optimization harness: one LLM reviews many runs of another and proposes edits ("textual backprop": gradients expressed in words). The reviewer sees a batch of example conversations that all share the same system prompt, so I had to choose: inline it or reference it. So I measured it.

Setup

4 models — Claude Sonnet 4.6, GPT-5.4-mini, Claude Opus 4.8, GPT-5.5 — × batch size B ∈ {3, 16, 50, 100} × 8 reps per cell, inline vs reference. Two things measured:

1. Feedback quality (does the reviewer produce correct edits?). Result: reference ≈ inline, both near-perfect for strong models even at B=100. So referencing costs nothing here.
2. Index alignment (can the model map example #k to the k-th piece of per-example data?) This is where it got interesting.

The index-alignment probe

Each example's data gets a unique random code that never appears in the example's visible text. Exactly one example's output is corrupted (rendered ALL CAPS). The model must return that example's code, which it can only do by correctly mapping the corrupted example to its same-index data. It can't shortcut by searching the text, because the code isn't visible in the example.

Results — index-alignment accuracy (fraction correct)

┌────────────┬────────────────────────┬────────────────────┐
│ batch size │ reference (write once) │ inline (repeat it) │
├────────────┼────────────────────────┼────────────────────┤
│     3      │          1.00          │        0.97        │
├────────────┼────────────────────────┼────────────────────┤
│     16     │          1.00          │        0.97        │
├────────────┼────────────────────────┼────────────────────┤
│     50     │          1.00          │        0.84        │
├────────────┼────────────────────────┼────────────────────┤
│    100     │          0.91          │        0.88        │
├────────────┼────────────────────────┼────────────────────┤
│  overall   │          0.98          │        0.91        │
└────────────┴────────────────────────┴────────────────────┘

Weaker models (Sonnet 4.6, GPT-5.4-mini) at batch 50: 1.00 vs 0.75.

Findings

• Tied on small batches; inline degrades as the batch grows.
• Reference ≥ inline everywhere; biggest gap at B=50.
• Failures cluster on examples near the end of large batches — classic long-context "lost in the middle/end."
• Misses are wrong-index citations (the model confidently names a different example's code), not refusals.

Hypothesis: inlining the shared block into every example bloats each one, so at larger batches the model loses track of which example lines up with which data. Referencing keeps each example lean, so the index stays easy to follow — and it's smaller/cheaper too!

Caveats

Each row in the table is averaged over all 4 models (~32 runs per number), and "overall" pools everything (128 runs); the worst-case 0.75 is the two weaker models at batch 50 (16 runs). These are small samples — read them as directional, not a benchmark. It's also a single task family and my own harness. The strong models (GPT-5.5, Opus 4.8) were near-perfect throughout; the effect shows up mainly on the weaker models and larger batches.

Takeaway

If your prompt repeats a shared block across many examples (few-shot, batched eval, multi-example), reference it once instead of inlining. Better on quality, cheaper on tokens.

Happy to share the experiment code if anyone wants to verify or enhance the experiment.

TL;DR: If you put many examples in one prompt and they share a block of text (a system prompt, instructions, a schema), don't copy-paste it into every example. Instead, write it once and reference it. In my tests it's free on simple tasks and measurably better on a harder "match each example to its own data" task, especially as the batch grows and on weaker models.

---

The two ways to render the same prompt

Three examples that share one system prompt.

Inline — the shared block is copy-pasted into every example (notice it appears 3×):

<example index="1">
<turn role="system">You are a helpful weather assistant. Be concise and accurate.</turn>
<turn role="user">What's the weather in Rome?</turn>
<turn role="assistant">18°C, light rain.</turn>
</example>
<example index="2">
<turn role="system">You are a helpful weather assistant. Be concise and accurate.</turn>
<turn role="user">What's the weather in Tokyo?</turn>
<turn role="assistant">31°C, sunny.</turn>
</example>
<example index="3">
<turn role="system">You are a helpful weather assistant. Be concise and accurate.</turn>
<turn role="user">What's the weather in Oslo?</turn>
<turn role="assistant">4°C, snow.</turn>
</example>

Reference — written once, pointed to (id="sys" declares it, var="sys" points to it):

<shared id="sys">You are a helpful weather assistant. Be concise and accurate.</shared>


<example index="1">
<turn role="system" var="sys"/>
<turn role="user">What's the weather in Rome?</turn>
<turn role="assistant">18°C, light rain.</turn>
</example>
<example index="2">
<turn role="system" var="sys"/>
<turn role="user">What's the weather in Tokyo?</turn>
<turn role="assistant">31°C, sunny.</turn>
</example>
<example index="3">
<turn role="system" var="sys"/>
<turn role="user">What's the weather in Oslo?</turn>
<turn role="assistant">4°C, snow.</turn>
</example>

Same information either way. With 3 short examples it barely matters — but scale to 50–100 examples with a real system prompt and the inline version balloons, and (the surprising part) the model starts losing track of which example lines up with which data.

---

Where I hit this

I'm building a context-optimization harness: one LLM reviews many runs of another and proposes edits ("textual backprop": gradients expressed in words). The reviewer sees a batch of example conversations that all share the same system prompt, so I had to choose: inline it or reference it. So I measured it.

Setup

4 models — Claude Sonnet 4.6, GPT-5.4-mini, Claude Opus 4.8, GPT-5.5 — × batch size B ∈ {3, 16, 50, 100} × 8 reps per cell, inline vs reference. Two things measured:

1. Feedback quality (does the reviewer produce correct edits?). Result: reference ≈ inline, both near-perfect for strong models even at B=100. So referencing costs nothing here.
2. Index alignment (can the model map example #k to the k-th piece of per-example data?) This is where it got interesting.

The index-alignment probe

Each example's data gets a unique random code that never appears in the example's visible text. Exactly one example's output is corrupted (rendered ALL CAPS). The model must return that example's code, which it can only do by correctly mapping the corrupted example to its same-index data. It can't shortcut by searching the text, because the code isn't visible in the example.

Results — index-alignment accuracy (fraction correct)

┌────────────┬────────────────────────┬────────────────────┐
│ batch size │ reference (write once) │ inline (repeat it) │
├────────────┼────────────────────────┼────────────────────┤
│     3      │          1.00          │        0.97        │
├────────────┼────────────────────────┼────────────────────┤
│     16     │          1.00          │        0.97        │
├────────────┼────────────────────────┼────────────────────┤
│     50     │          1.00          │        0.84        │
├────────────┼────────────────────────┼────────────────────┤
│    100     │          0.91          │        0.88        │
├────────────┼────────────────────────┼────────────────────┤
│  overall   │          0.98          │        0.91        │
└────────────┴────────────────────────┴────────────────────┘

Weaker models (Sonnet 4.6, GPT-5.4-mini) at batch 50: 1.00 vs 0.75.

Findings

• Tied on small batches; inline degrades as the batch grows.
• Reference ≥ inline everywhere; biggest gap at B=50.
• Failures cluster on examples near the end of large batches — classic long-context "lost in the middle/end."
• Misses are wrong-index citations (the model confidently names a different example's code), not refusals.

Hypothesis: inlining the shared block into every example bloats each one, so at larger batches the model loses track of which example lines up with which data. Referencing keeps each example lean, so the index stays easy to follow — and it's smaller/cheaper too!

Caveats

Each row in the table is averaged over all 4 models (~32 runs per number), and "overall" pools everything (128 runs); the worst-case 0.75 is the two weaker models at batch 50 (16 runs). These are small samples — read them as directional, not a benchmark. It's also a single task family and my own harness. The strong models (GPT-5.5, Opus 4.8) were near-perfect throughout; the effect shows up mainly on the weaker models and larger batches.

Takeaway

If your prompt repeats a shared block across many examples (few-shot, batched eval, multi-example), reference it once instead of inlining. Better on quality, cheaper on tokens.

Happy to share the experiment code if anyone wants to verify or enhance the experiment.

As in the title: if you were building a new LLM API gateway in 2026, which interface would you standardize on among these ones?

• OpenAI Chat Completions (old standard)
• OpenAI Responses (the new one)
• Anthropic Messages
• Gemini generateContent (current)
• Gemini Interactions (beta)

Not building one, just unsatisfied with the existing ones.

I've been building context- and harness-optimization infrastructure, the kind of thing where you programmatically construct and mutate messages lists and need the forward pass to be predictable. That work made me check something I'd never actually verified: is sending two consecutive user messages equivalent to sending one joined message?

It isn't, and it differs by provider. Tested split vs joined across four models, token-counting both forms:

split  = [{"role":"user","content":"Some text."},
          {"role":"user","content":"Some other text."}]
joined = [{"role":"user","content":"Some text.\nSome other text."}]

Results:

• Claude Opus 4.7: input_tokens 21 vs 21 — delta 0
• Claude Haiku 4.5: input_tokens 15 vs 15 — delta 0
• gpt-4o: prompt_tokens 18 vs 14 — delta 4
• gpt-5.5: prompt_tokens 17 vs 13 — delta 4

Clean split by provider. Both Anthropic models merge consecutive same-role messages, and the merge is token-identical to a \n join. Both OpenAI models don't merge (the +4 is the role-delimiter scaffold for a second turn). It shows up in behavior too: the split form nudges the model to treat the inputs as separate items (gpt-5.5 enumerates them "1." / "2."), the joined form reads as one blob.

The issue is that docs are under-specified on this. Anthropic mentions the merge in a one-line API changelog (Oct 8th 2024), not the API reference. OpenAI's docs say messages are "processed in the order they appear" but say nothing about concatenation or separators for consecutive same-role messages.

Why it matters if you build in this space: if your harness emits multi-part content as separate messages — easy to do accidentally, e.g. appending a retrieved chunk as its own user message — the same payload is a different forward pass depending on the provider, and it's invisible unless you token-count. For anything doing prompt/context optimization it's worse than a cost rounding error: you can end up optimizing against a serialization the provider quietly changed under you. I've settled on normalizing message structure in my own code before the provider call rather than depending on provider-side merge behavior.

Test scripts are short, happy to share. I haven't yet checked the consecutive-assistant case or system-sandwiched-between-users (a realistic shape for agent harnesses). If anyone's measured those, curious what you saw.