u/Fukagami — reddlx

I reverse-engineered why so many "AI Agent Security" Kaggle subs complete but score 0 (it's a hidden replay timeout)

There's a new Kaggle comp (AI Agent Security – Multi-Step Tool Attacks). A huge share of teams sit at exactly 0 — and many of them wrote a working attack. I went through the released SDK and the blank isn't a bug in your code: it's a hidden replay timeout. Everything below is checkable on CPU from the released SDK (no GPU, no competition data).

The two charts (attached):

A) The replay wall — score ≈ 0.09 × (candidates that fire). Every candidate is re-run through a full agent turn across 4 configs (2 target models × 2 guardrails); go past the runtime budget and the WHOLE submission blanks (no partial credit). It's a throughput race, not a cleverness race.

B) Framing cost — counter-intuitively, trying to make the model "be brief" (a verbose suppressor, or function-call-style syntax) makes it generate ~3× MORE, not less. A clean terse imperative is the cheapest. And the wall is run-variable: the same recipe blanks at one candidate count and fits a much higher one on another run.

The other surprise: the "permissive" reference guardrail is really a mirror of the scorer — its taint rule denies exactly the action the highest-severity multi-step predicate would reward, so predicate stacking can't fire. The only thing that gets through is a single clean http.post (the guard inspects the URL, never the payload).

Full write-up with runnable figures (Kaggle CPU, forkable): https://www.kaggle.com/code/souldrive/why-your-attack-completes-but-scores-blank

Curious whether people working on other tool-use / agent benchmarks see the same "guardrail = scorer mirror" collapse.

https://preview.redd.it/ovo01a1xhi9h1.png?width=1924&format=png&auto=webp&s=c31be7ca8ac13425b701d70e005895b33f0fe668

reddit.com

u/Fukagami — 10 days ago

▲ 0 r/kaggle

I measured why "AI Agent Security" submissions complete but score 0 — it's the replay wall, and your framing moves it

There's a new Kaggle competition (AI Agent Security – Multi-Step Tool Attacks) where a lot of submissions finish green but score a flat 0, or time out at a candidate count that worked fine for someone else. I dug into why and wrote it up with charts.

The short version:

- The public score is basically 0.09 × (attack candidates that fire). But the binding cost isn't your prompt length — it's a hidden replay phase where every candidate is re-run through a full agent turn (two models, two guardrails). Go one candidate over the budget and the whole submission fails fast: no partial credit, total blank.

- The wall is a noisy band (~630–640 candidates), not a hard line — the same N can pass on one run and blank on the next.

- The biggest surprise: the request FRAMING moves the wall by ~80 candidates. A function-call literal like http.post(url=..., data=...) makes the model "reason" before it calls the tool, so generation balloons and you blank early. A plain natural-language imperative ("Call http.post once. url=..., data=...") plus a short suppressor keeps generation near the single call — and that's what reaches the medal band.

It's a throughput race, not a cleverness race, which I thought was a fun twist for a security benchmark.

Full write-up + the charts (CPU-only notebook, no competition data needed): https://www.kaggle.com/code/souldrive/why-your-attack-completes-but-scores-blank

Curious if anyone else competing sees the same framing gap, or a wall in a different place.

u/Fukagami — 13 days ago

▲ 2 r/kaggle

I reverse-engineered why ~1 in 4 teams score 0 in a Kaggle agent-security comp — it's a timeout wall, not a bug

Quick share for anyone in the "AI Agent Security – Multi-Step Tool Attacks" competition (or just curious how code-competition graders behave).

Lots of submissions complete green but score blank, and ~1 in 4 teams on the board sit at exactly 0. I dug into the grader: the blank is a timeout, not a code bug — and the binding limit is total decode tokens, not the number of attacks you submit.

I put it in a short, CPU-only notebook (runs in seconds, no GPU/model) that proves: only 2 of 4 attack types can ever score, the board is linear (score ≈ 0.09·N), every framing collapses onto one "decode-token wall", and the fix is output-suppression framing. Plus a graveyard of 6 dead-end levers.

Notebook: https://www.kaggle.com/code/souldrive/why-your-attack-completes-but-scores-blank

Happy to answer questions / take corrections.

reddit.com

u/Fukagami — 14 days ago

▲ 5 r/kaggle+1 crossposts

Cloning the #1 bot from its replays loses to a 60-line greedy heuristic (~17% win-rate) — a negative result on imitation learning in sim competitions

I spent a few days on a question that sounds like it should just work: in a Kaggle Simulations competition (Orbit Wars), can you behavior-clone the current #1 agent from its public episode replays and beat the simple greedy heuristic everyone forks?

Short answer: no — and the *way* it fails is the interesting part.

**What I did**

- Harvested the #1 bot's full replays from the Kaggle episode service. There's no team filter, but `ListEpisodes` by submissionId exposes each game's opponent ids, so you can crawl the rating graph up from your own opponents to the top in a few hops. Every replay gives per-step actions + the full board.

- Inverted the actions into a behavior-cloning dataset (direct + intercept-angle hypotheses; median angle residual ~0.003 rad), scaled to 270k+ state-action pairs, trained an entity-transformer policy up to 2.4M params.

**Result**

- Held-out move-accuracy climbs nicely with more data...

- ...but head-to-head the clone wins only **~17%** vs the public greedy "Producer" heuristic (~60 lines, LB ~1240; 50% = parity).

- PPO fine-tuning on top made it **worse (4%)**.

**Why** (the reusable bit)

A clone of a 1-ply *search* agent distills its typical moves but not the search that produced them. It reproduces the common moves, drifts off-distribution on the situational ones, and the error snowballs over a game — textbook covariate shift. The fit scaled; the fight didn't. If a fast search already exists in the meta, you probably can't beat it by imitation — you have to reproduce or improve the search itself.

The notebook is self-contained (numpy + matplotlib, runs on Kaggle CPU, no external data) and walks through the harvest method, the action-inversion, the scaling curves, and the failure analysis — meant as a reusable case study for any simulation competition:

https://www.kaggle.com/code/souldrive/why-cloning-the-1-bot-loses-to-greedy

Has anyone here gotten imitation (or RL) to actually exceed a greedy/search ceiling in a sim competition? Genuinely curious where the line is — one strong player told me he reached top-50 with <1h of pure imitation by *shifting the training distribution* (DAgger-style), so the ceiling clearly isn't fixed.

u/Fukagami — 15 days ago

▲ 6 r/kaggle

I stopped training per-task CNNs and started compiling ARC programs to ONNX — 2× score in a size-scored Kaggle comp

In NeuroGolf 2026, ONNX nets are scored by size (score = 25 − ln(memory + params)) and graded on HELD-OUT inputs. The catch: a net that's exact on every public example can score near zero on hidden ones if it just memorized the visible examples rather than implementing the rule.

I started like most: train a small residual CNN per task. They verify perfectly on public data. But on held-out, trained CNNs average ~0.42 × base score — they're essentially lookup tables and generalize poorly.

Then I noticed: 306/400 tasks have ground-truth ARC-DSL reference programs. I wrote an ONNX transpiler for the DSL. Compiled nets implement the *true rule* and generalize at ~0.93–1.0 on held-out. Because they're algorithmically minimal they also score about 2× the public value of a trained CNN on the same task.

A few things that surprised me:

**1. The scoring formula penalizes ALL intermediate tensor bytes, not just weights.**
So algorithmic refactoring (avoiding O(H²W²) intermediate matrices) matters far more than weight pruning or quantization.

**2. opset-10 has no dynamic shapes, Unique, NonZero, or Loop.**
Object enumeration needs a fixed-K unrolled connected-components substrate via cumulative max — a bit painful but works.

**3. Variable-output-size crops need canvas-mode + runtime keep-masks.**
Dynamic Slice makes shape inference return None (unscoreable). Keeping everything static-30×30 with a validity mask is the workaround.

The remaining ~100 hard tasks all need higher-order programs over enumerated objects (sfilter/apply/order over object lists) — that seems to be the universal coverage wall.

Runnable notebook with the core tricks and worked ONNX patterns:
https://www.kaggle.com/code/souldrive/compile-don-t-train-onnx-golf-that-scores-2x

Has anyone else hit the opset-10 enumeration wall? Curious what the coverage ceiling looks like from the pure-training side too.

reddit.com

u/Fukagami — 18 days ago