u/Reasonable_Royal_621

ok so quick context if you haven't seen the other posts: I've been messing around with CacheVerifier, basically testing whether bolting a verifier onto semantic caching actually helps. right now it's dumb and binary, candidate answer either gets a thumbs up or thumbs down, no in-between.

there's this other paper, TweakLLM (arXiv:2507.23674), that does something I think is genuinely smarter: instead of rejecting a bad candidate and eating the full regen cost, it has a cheap LLM just... rewrite the candidate so it fits the new query. patch it instead of throwing it out. I want to add that as a comparison to my own setup and I've been stuck on it for a while, so figured I'd just ask here, since this sub has already bailed me out twice on this project (the axis-problem theory and the bucketing design both came from comment threads here, not from me).

here's where I'm stuck. everything I currently measure is trace-based against public benchmarks , "was this correct" comes entirely from the dataset's own labels, no actual LLM judge anywhere in the loop. works great when the answer is binary. falls apart completely once you're rewriting text, because now you've got a brand new string that isn't in any label anywhere. nothing to check it against.

things I've considered and don't love:

just throw an LLM judge at grading the rewrites. but now I'm introducing a whole new cost/noise source that literally nothing else in this project needed, and "let an LLM grade another LLM's output" is its own whole mess
when there happen to be multiple reference answers for the same query cluster, score the rewrite against one of them by similarity. except that's literally the "similarity ≠ correctness" problem this entire project exists to complain about. using it as my metric here feels like cheating on my own thesis
just skip fine scoring, measure something crude like "did rewriting recover some recall vs just rejecting," and not even try to put it on the same hit-rate/error-rate curve as everything else. doable but honestly a weaker result than I want
if anyone's had to evaluate a generate-a-rewrite step where there's no clean ground truth for the output, not classification, not ranking, an actual freeform string you have to judge somehow , genuinely curious how you dealt with it. or if you think I'm overcomplicating this and should just pick one of the above and move on.

repo's here if you want the full context on what's been tested so far: https://github.com/imxinchengyou/CacheVerifier

I spent weeks calling my reproduction of a published semantic-caching baseline (vCache's adaptive-threshold policy) "faithful" because every formula matched their paper exactly. It wasn't. The bug was two rows of fake data in their source code that never made it into the paper, and fixing it raised hit rate by 4x to 29x depending on the dataset.

I'm running a research project (CacheVerifier) comparing a synchronous verification mechanism for semantic LLM caches against a couple of published baselines, one of which is vCache's adaptive-threshold policy. A few weeks ago I ported that policy — read their paper's algorithm description, then went through their actual source and matched every formula: the logistic regression design matrix, the gamma clipping, the delta-method variance, the perfectly-separable-case variance table (copied their exact lookup values), the tau grid search, all of it. Formula by formula, it checked out. I was confident enough to write "faithfully ported" in the paper and move on.

Today I finally did the thing I should've done from the start: cloned vCache's actual repo and diffed my port against the real running code, not just the formulas I'd extracted from it. Everything still matched — except one class I hadn't looked closely at, the one holding each cache entry's observation history.

Their constructor does this:

self.observations: List[Tuple[float, int]] = []
self.observations.append((0.0, 0))
self.observations.append((1.0, 1))

Two fake observations, baked into every single cache entry the moment it's created, and never removed. A "similarity 0.0 → wrong" and a "similarity 1.0 → correct," permanently sitting in the history feeding every logistic regression fit for that entry's whole life.

My port started from an empty list. Nothing malicious, no misreading of any formula — I just didn't know these two rows existed, because they're not mentioned anywhere in the paper, only in the source.

Here's why it actually matters and isn't just a cosmetic difference: the algorithm needs 6 observations before it'll ever trust an entry enough to serve it from cache (min_observations=6, this part is in the paper). With two observations already pre-loaded, their implementation only needs 4 real ones to clear that bar. Mine needed the full 6. Every entry in my version sat in cold start two observations longer than the real algorithm, every single time.

Fixed it (one line, empty list → [(0.0, 0), (1.0, 1)]) and reran the full thing on all three datasets I test on. Hit rate went up everywhere — between 4.4x and 29.1x depending on dataset and target error rate. Best case, one dataset at the tightest error budget: 0.04% → 1.21%. And the part I actually care about most: error rate stayed under the target ceiling at every single point I checked. The algorithm's formal guarantee was never violated by my bug — I just wasn't letting it do nearly as well as it's designed to.

So for weeks I had a "faithful reproduction" that was quietly making a competing algorithm look almost useless (fractions of a percent hit rate), when the actual bottleneck was two rows of bootstrap data I'd never have found by re-reading the paper one more time, only by diffing the real code.

If you're reproducing someone else's algorithm as a baseline for a comparison — not approximating it, not "inspired by," but claiming to faithfully port it — matching the published formulas is necessary and not sufficient. Constructors quietly seed state that never makes it into the paper. Go clone the actual repo and diff against it, not just the pseudocode. I got lucky that I decided to check at all.

Repo's got the before/after numbers if you want to see the full breakdown: https://github.com/imxinchengyou/CacheVerifier

Follow-up to the axis-problem / bucketing post from a few days ago. Said I hadn't run it yet and welcomed holes being poked in the design. Ran it. Numbers didn't cooperate.

Setup was what post 2 described: extract the action verb from both the query and the matched candidate (rule-based, spaCy dependency parse + lemmatizer, not embeddings), restrict the verifier to pairs where the extracted actions match, see if AUC goes up on that restricted subset compared to the verifier running on everything.

On SearchQueries: verifier-only AUC on the extraction-eligible pairs was 0.6215. Restricted to bucket-matched pairs only, it was 0.6150. Not higher. Slightly lower, though the confidence intervals basically overlap, so call it a wash rather than a regression.

A few things worth flagging honestly:

only 27.3% of gray-zone pairs had a cleanly extractable single action verb on both sides. Most pairs just didn't qualify for this mechanism at all.
bucket match by itself, used as a standalone yes/no predictor with no verifier involved, had a 45% false-approve rate. Not great as a filter on its own.
Pulled some real examples to see what's actually happening in each failure mode, not just staring at the summary numbers:

Most of that 73% extraction failure is just... short queries with no verb. "best vegetarian restaurants near me" vs "best vegan food near me" — that pair is a real gray-zone match, genuinely a different topic (vegetarian ≠ vegan), and my extractor has nothing to say about it because neither string has a verb to grab onto. A lot of search-query traffic just looks like this.

Then there's a parsing failure I didn't anticipate: "do you have to pay to charge an electric car" got tagged with action='have' (the auxiliary), while "cost to charge electric car" got action='charge' (the real one). Different roots, so bucketing calls it a mismatch and throws the pair out — except it's actually a correct match. The dependency parser grabbed the wrong verb, not because of some deep semantic issue, just a parsing artifact on an auxiliary-heavy phrasing.

And the one that actually undercuts the theory a bit: "convert audio to video youtube" vs "convert youtube video to audio" — same action, 'convert', on both sides. Bucket says match. They're opposite operations, and ground truth agrees this pair is wrong. Action-verb matching alone doesn't see that the object got flipped — you'd need the object, not just the verb, to catch this one.

So the theory (cosine similarity has no axis for the operator/action, bucket on it first) is still logically sound as an idea, but "extract the action verb" turned out to be a narrower net than I expected: no verb to grab in most short queries, occasional wrong-verb extraction on auxiliary-heavy phrasing, and even when it works cleanly it doesn't see swapped objects. Any one of those alone might not sink it. All three together seem to be enough to wash out the AUC gain the theory predicted.

Repo's updated with the raw numbers: https://github.com/imxinchengyou/CacheVerifier

Posting the null result because burying it felt worse than the result itself. If anyone's got a read on why this didn't move, happy to hear it before I go digging.

Okay so remember that post about semantic caching confidently serving the wrong answer ("cancel my subscription" → cached answer for "pause my subscription," 0.87 similarity, dead wrong)?
Full thing's here: https://github.com/imxinchengyou/CacheVerifier. Quick recap for anyone landing fresh: tested whether a verifier model beats just tuning the threshold.

Y'all had a really good theory in the comments: cosine similarity encodes topic, not which action word, so nothing separates "cancel X" from "pause X" no matter how you tune the number. And that this was specifically why my SearchQueries benchmark tanked while a longer-text benchmark was fine.
So I actually built the experiment to test it properly — controlled pairs, fixed-object-varied-action vs. fixed-action-varied-object, long and short phrasing, the whole thing. And it just...didn't hold up. Similarity signal half-agreed, verifier signal flat-out disagreed, and the "independent of length" part wasn't independent of length at all. A good theory, tested, dead.Normally that's the end of the post — "welp, guess we don't know," sad trombone.

Except building that experiment made me actually stare at the raw SearchQueries data instead of trusting the summary stats, and that's when I found the dumbest possible explanation sitting right there the whole time: all 150,000 "answer" fields in that dataset were the exact same string. Not real answers. Just "Not required for the benchmark because of the id_set", copy-pasted 150,000 times, straight from the public benchmark release. My verifier had been scoring (real query, the same 11 words every single time) for the entire experiment. It's not that the model couldn't tell cancel from pause，it never even got to look at an answer.

No axis theory needed. Just a spreadsheet cell that never got filled in.

Regenerated real answers, reran everything, and the two halves of the story moved in opposite directions, which I did not expect:
- Fine-tuned verifier got even better than I originally reported (46/54 → 53/54 wins against just tuning the threshold).
- Off-the-shelf verifier got worse — turns out it wasn't harmlessly clueless, it was confidently wrong often enough to actively hurt you (loses outright at 23/36 points).

Full erratum's up, kept the old broken result files in the repo too so anyone can diff them: https://github.com/imxinchengyou/CacheVerifier.

Mostly just wanted to share the debugging story because I think it's a useful cautionary tale — the fancy semantic explanation was plausible, well-argued, and worth testing, and it still wasn't it. Sometimes the boring, embarrassing answer (check your data before you check your theory) is the right one. Thanks for pushing me to actually test the interesting version instead of letting me publish a vibes-based conclusion.