u/COAGULOPATH

Sorry for the strange question...

In Stephen King's biography On Writing, he relates the following:

>While he was going to college my brother Dave worked summers as a janitor at Brunswick High, his old alma mater. For part of one summer I worked there, too. (...) I got paired with a guy named Harry, who wore green fatigues, a big keychain, and walked with a limp. (...) One lunch hour Harry told me what it had been like to face a Japanese banzai charge on the island of Tarawa, all the Japanese officers waving swords made out of Maxwell House coffee cans, all the screaming enlisted men behind them stoned out of their gourds and smelling of burned poppies. Quite a raconteur was my pal Harry.

Is there any historical corroboration of this sort of thing (makeshift swords made of coffee cans)?

It seems implausible: how did King's friend know the swords were made from Maxwell House coffee cans, specifically? There's Youtube videos of people making horrible "swords" from soda cans, but this involves melting the metal, which would obviously remove any identifiable branding or marking.

I have looked and can't find anything (and weird/funny novelty weapons typically have a large footprint online). Is there any evidence of stuff like this, or is it a tall tale?

Thoughts? Any stories that stand out?

edit: super high effort OC I know. Will post thoughts on some of them when time becomes available.

I have lost about 10 times in a row, always around level 15-25.

I inevitably fall in a failure spiral: my Pokemon die, fall behind in levels because they're dead half the time, they die even more, I run out of money for revives and heals, etc.

I'm finding this significantly harder than Classic or Endless. When I lose, I don't even understand what I could have done differently to win.

Do you just hope you RNG into an amazing TM or item that carries you? Or is there a known strategy/starter for beating this?

edit: I would prefer not to cheese by reloading savestates.

A puzzling issue: given strong LLM truesighting ability (Opus can frequently identify the author of unpublished, unseen text), shouldn't they be strong AI detectors? GPT-4o alone has contributed OOMs more text to training datasets than any one human: if there was any author they could truesight, wouldn't it be themselves?

(...unless maybe the sheer amount/diversity of LLM-generated text hurts rather than helps at a certain point, like if the footprints at a crime scene also tracked through every house in town. But humans can often learn to spot LLM-generated text—some even learn to recognize tells from certain models, eg "delve" = older GPT-3.5/4, "Sarah Chen" = Claude. So why do LLMs struggle to do the same?)

According to Pangram, apparently they now do it fairly well.

2022/2023 models like GPT-4 cannot distinguish LLM text from human text at all 0-shot, for reasons that seem obvious.

Once GPT-4 is seeded with examples of what AI text looks like, its scores rise to 85%, similar to 0-shot performance of today's models.

Obviously a 15% error rate (or even GPT 5.5's 5%) is unacceptable if you care about false positives.

(And this is still far less ability than I'd expect: if LLMs can clock Kelsey Piper from decades-old school reports that she's never published online, why can't they reliably tell you the endpoint for a given piece of text: "ah, yeah, this is Kimi-k2-6" or whatever? Why is their limit apparently "AI or not AI"?)

An interesting side topic: how do LLMs differ in their ability to evade AI detection?

A year back I generated some slop, ralphed 5x with "rewrite to make this look human-written by adding spelling/grammatical errors and unusual word choices", and Pangram still detected it as AI generated. Obviously not a great test.

In May, it was reported that a number of FrontierMath problems had mistakes in them that made them technically unanswerable, and top LLM scores were likely depressed because of this.

This issue turned out to be way worse than I thought. They have released a new version of the benchmark that addresses errors in 42% (!) of questions.

Most LLM scores have greatly shot up, often by 1.5x or more.

The current highest score is Claude Fable, at 88% (they're still re-testing some of the GPT-5 Pro models). This is on the Tier 4 dataset.

All benchmarks have some number of bad questions that can't be answered (I think the MMLU had about 5-8%). But this is extremely egregious.

Also, there are likely still more errors to be found. Hard to know how else to explain Fable scoring lower on Tiers 1-3 than Tier 4 (which is supposed to be the hardest...)

>Today’s coding benchmarks have established that models can write correct code. But as AI-generated code becomes the dominant path to production, correctness is now table stakes. The question that we should be asking is: can models actually write good code?

>We’re excited to introduce FrontierCode, a benchmark that measures how well models can truly meet the standards of high-quality production codebases. What sets us apart:

>

>Our benchmark provides the strongest available signal of a model’s ability to write high-quality, maintainable code. We find that even today’s most capable models struggle on this new standard.

This is by Cognition, the creators of early 2024 coding agent Devin.

It looks interesting, though the graphs have some suspicious results (Opus 4.8 scoring 2.5x better than Opus 4.7, models degrading as more test-time is used).

What are some complete shitmons that remain bad even with PR's abilities and egg moves?

I'll start: Claydol. What does it even do? It has 70/70 attack, is slow (yet too fast for trick room), kinda has the Hoenn special (100/120 defenses feel like nothing when you're Psychic), has a movepool like a barren desert, etc...it's truly one of the most unfortunate Pokemon ever conceived.

Well-Baked Body and Levitate put it in the fabled "triple immunity" club, but this is almost wasted since ground and fire attacks wouldn't bother it much anyway.

Claydol actually formed part of one of the first teams I used to complete Classic. My main had Parabolic Charge and I needed an electric immunity, so I had Claydol sitting uselessly in slot 2, doing what it does best: nothing.

This paper does a bunch of stuff, mostly concerning the creativity of LLM text. What they find:

- Training models to maximize n-gram novelty adds textual variety, but damages coherence (or "pragmaticality", as they term it). The tails come apart. "While n-gram novelty is positively associated with expert writer-judged creativity, approximately 91% of top-quartile n-gram novel expressions are not judged as creative."

- LLMs can be used as judges, and track pretty well with human experts when predicting novelty. (p9) But they struggle to match human judgment when identifyng coherency/pragmaticality issues.

These two issues (the second more than the first IMO) may explain certain failure modes in current LLMs.

Capabilities have raced far ahead on grading creativity vs grading coherence (likely because it's a harder task: a sentence's novelty can be judged in isolation, but coherence also requires knowledge of the full surrounding context) so we get a bias toward weird, florid "poetic" text that doesn't make sense (GPT-5 generated example they provide: "[person] said the morning blessings in a whisper that embarrassed the chairs".)

Other findings:

- Small models suck.

- Fine-tuning and few-shot doesn't appear to do much.

- Scalar reward models seem promising.

It's fast and the benchmarks look good, mostly surpassing Gemini 3.1 Pro. But it's very expensive for a Flash model.

>It achieves speeds of over 280 output tokens/s, but higher token usage and token pricing make it over 5x more costly to run the Intelligence Index than Gemini 3 Flash, and 75% more costly than Gemini 3.1 Pro. Gemini 3.5 Flash is $1.50/1M input and $9/1M output tokens, Gemini 3 Flash was $0.5/$3 per 1M input/output tokens, a 3x increase. The rest of the increase was driven by higher token usage when running our benchmarks

This model is apparently now powering AI Mode in Search, which surprises me: I wouldn't have thought that would be affordable.

Also:

>We’re also hard at work on 3.5 Pro. It's already being used internally, and we look forward to rolling it out next month.

Both models spent $10,000 (the limit). GPT-5.5 scored 0.4% and Opus 4.7 scored 0.2%.

This benchmark is quite difficult for clankers. It seems almost pointless to test current LLMs on it: they all score equally (about zero). My prediction of a 30% score in a year seems unlikely to come true.

It's probable that new breakthroughs (or at least much better base models) are needed here. (That said, when LLMs finally do chip a dent in ARC-AGI-3, even a little one, expect scores to shoot to 100% quite fast)

So far, so boring.

Less boring is the ARC Prize's analysis of how GPT-5.5 and Opus 4.7 played, based on reasoning from 160 games. The two models failed in extremely unlike ways.

Opus 4.7 aggressively theorycrafts, and learns game mechanics fairly well. But it assumes facts not in evidence, struggles to integrate new data into existing beliefs, and often can't (or won't) backtrack out of wrong assumptions. It ends up playing from a theory of the game that is "neat, plausible and wrong."

GPT-5.5 just...doesn't commit to a theory. Ever. It taps buttons but never seems to learn anything. In every turn, it sounds like an old man who has woken from a deep slumber and is seeing the game for the first time ("I'm analyzing a game with a grid..."). It blindly wonders if it's playing Tetris, or if the orange blocks are lava. Everything gets pattern-matched onto some existing videogame, with its previous reasoning forgotten.

It's funny that GPT-5.5 "doubles" Opus 4.7's score. To the extent this isn't noise, it's likely due to GPT-5.5's exploration-focused approach getting luckier a little more often.

tldr: Opus 4.7 is precise but inaccurate, GPT-5.5 accurate but imprecise.

Do tests like ARC-AGI-3 mean much, in the end? I'm not sure. I suspect the games were designed (in part) to focus around things that humans find easy and LLMs find hard, like spatial reasoning. But many important things (like robotics) involve spatial reasoning: I see this as defensible.

(I got around 80% on the two games I played. According to its creator, "Any smart human giving it real effort should score >90% on ARC-AGI-3". y u bully me man :( )