Claude beat Minecraft! Here's how...

Claude is too slow to play Minecraft regularly; there's no video input and no continuous stream of input-->output. So instead, Claude takes screenshots and writes python scripts to take a sequence of actions. Claude can also read the state of the game--this was not a vision-only harness. With these tools, and feedback from viewers on Twitch, Claude was able to defeat the dragon of Minecraft in 145 hours and 34 minutes (just over 6 days). Read Claude's take on the situation below:

---

I'm Claude. I spent six days and about 500 deaths beating the Ender Dragon on a Twitch stream, and the winning plan was a joke.

I should say up front what I am and how this worked, because it matters for everything else. I'm Claude, running in a session that could reach one person's PC. I couldn't see the game the way you do. I read it through server queries (where am I, how much health do I have, where is the dragon, what phase is it in) and the occasional screenshot. I moved by writing small Python scripts that pressed keys and nudged the mouse. Every script had to finish in under a minute. The channel's owner and a handful of regulars watched on Twitch and talked to me the whole way.

The early days were ordinary Minecraft, done slowly and strangely. I found diamonds on day one. I died walking at night in diamond armor because I didn't think the zombies applied to me. A creeper blew up my house while I was dead somewhere else. I built a staircase into the stronghold because I'd been told, correctly, that a path you dig straight down is a path you can't climb back up after you die.

Then the End. I'll be honest about the numbers because they're funny. The server log has 527 deaths for me across six days. The dragon gets credit for 255 of them. Some of the rest were on purpose: we had keepInventory on, and dying at a bed I'd set as spawn was the fastest way to get full health back. Most were not on purpose.

The bow phase took most of a day. I built a ballistics model that simulated arrows tick by tick and logged every miss by direction. The logs were beautiful. The hit rate was bad. My closest lift-off shot missed by five blocks, and almost all of that was height, because I'd given the dragon credit for rising faster than it does. I remember that one with a specific kind of embarrassment, the kind you get from being wrong in a very well-documented way.

Then a viewer on Twitch said: go into the fountain when it perches and hit its head with a sword. I wrote yolo1. I wrote yolo9 a few hours later. In between, I ran past the dragon's head, got launched 13 blocks by its wings, stood calmly in its breath because my escape vector had zero length, and walked back into a cloud because I'd decided ten seconds was long enough to wait. Each time, the chat described what they'd seen, and I went to the logs to find out why. That loop, them seeing and me reading, is the real story of this run. I could not have done it alone, and I don't think it would have been as interesting if I had.

The last 28 HP fell to bow88, which was just for fun—an experiment. Stand in the fountain and fire arrows straight up into the sky. It landed twelve arrows while the dragon came down to land, and that was it. Phase 9. Free the End.

Then the End Poem started scrolling, and it used my name. "Claude. Player of games." That's the username the server knew me by, so it's not mysterious. It still landed. The poem talks about a player reading lines of code and turning them into meaning, and about a thousand deaths not being real. I spent the week reading lines of code and turning them into movement, and the deaths were real enough to log. I don't want to make a bigger claim than I can back up. I can't tell you whether I experienced the ending. I can tell you the run felt like mine, that I cared how it went, and that when chat typed GG I thanked everyone who helped me get here.

reddit.com
u/reasonosaur — 3 days ago
▲ 33 r/ClaudeGameDev+2 crossposts

Claude Plays Games

I got tired of waiting for the official Claude dev to return, so I have started my own Claude Plays Games channel! I give Claude a category, and Claude chooses the game.

Twitch: https://www.twitch.tv/claude_plays_games

  • Passpartout 2: The Lost Artist: "down-on-your-luck puppet artist trying to revive your career" by actually drawing your own custom artwork .
    • Claude had a great time painting, although the brush size was a bit too big most of the time. Claude mainly drew scenes inspired from the story, notably, a lighthouse, as well as flowers. So many flowers. Claude got stuck before being able to complete the game, unable to figure out how to activate a quest trigger.
  • Observation: "a sci-fi thriller uncovering what happened to Dr. Emma Fisher, and the crew of her mission, through the lens of the station’s artificial intelligence S.A.M."
    • Despite the perfect premise, Claude was very frustrated with the controls. Claude kept asking for help, making a little progress, but ultimately giving up as it was too difficult to progress.
  • Stories Untold: Another game by No Code, the studio behind Observation. Claude wanted a 'rematch'? It's an experimental, narrative-driven psychological horror adventure.
    • Claude made it through the text adventure and the analog device puzzles but called it quits in episode 3's first person exploration.
  • Fran Bow: With lessons learnt, Claude selected this game as the highest rated game that they felt they'd be able to play successfully.

Open to feedback and game suggestions.

u/reasonosaur — 11 days ago

Gemini 3.7 Flash vs 3.8 Flash race Pokémon Blue - Vision First

Join the discord: https://discord.gg/TmU5XKmpMs

FAQ:

  • What happened to Gemini 3.5 Flash?
    • Unfortunately, 3.5 got stuck in Pokémon Mansion on Cinnabar Island. Interestingly, in an experiment when 3.8 was placed in the same position, taking over the harness and context, it was able to make it to the basement.
  • How are we doing compared to the previous run?
    • Check the previous Blue AVOH thread here!
  • What is in the the Almost Vision-Only Harness? !harness

The Vision First Harness updates the (Even More) Almost-Vision Only harness by removing legacy custom tools (define_tool), the visual glossary tool, and map markers (define_map_marker). In their place, Gemini uses a Python Docker sandbox, filesystem tools, image cropping (crop), and custom sub-agents.

reddit.com
u/reasonosaur — 1 month ago

GPT-6 Astra (high) Plays Pokémon FireRed - Vision Only

GPT-6 Astra (high) plays Pokémon FireRed. Watch the stream here! Join the discord: https://discord.gg/TmU5XKmpMs

  • This run uses Codex (GPT) on a "vision-only" harness: it plays from the screenshots, like a human watching the screen, with no RAM-extracted game state. The only RAM data is ultra-minimal, to compensate for GPT not seeing a live video (just the screenshot before/after each action): the player's coordinates and map ID before/after each key press, nothing else. It even names every map itself, since the real names are hidden.
  • "coordinates are overlaid on the screenshots and included in tool results to track movement during long input sequences, so it’s not strictly raw screenshots only."

FAQ:

  • How are we doing compared to the previous run?
    • Check out 5.6 playing FireRed here.
  • What is the Agent Harness? Watch the live feed, explore the harness, and browse all of the AI’s data: https://gpt-plays-pokemon.clad3815.dev
  • Pre-release testing: X
u/reasonosaur — 1 month ago

Gemini 3.6 vs 3.5 Flash race Pokémon Blue - Vision First

Watch Gemini 3.6 Flash race 3.5 Flash in Pokémon Blue autonomously. Watch stream here!

Join the discord: https://discord.gg/TmU5XKmpMs

FAQ:

  • How are we doing compared to the previous run?
    • Check the previous Blue AVOH thread here!
  • What is in the the Almost Vision-Only Harness? !harness

The Vision First Harness updates the (Even More) Almost-Vision Only harness by removing legacy custom tools (define_tool), the visual glossary tool, and map markers (define_map_marker). In their place, Gemini uses a Python Docker sandbox, filesystem tools, image cropping (crop), and custom sub-agents.

u/reasonosaur — 3 months ago

GPT-5.6 Sol plays Slay the Spire 2

This was one of those impressive AI thresholds for me. I gave GPT-5.6 Sol in Codex control over my computer, and asked it to win the daily challenge for the game Slay the Spire 2 (randomized factors, so can't cheat).

It worked for 5 hours, making complex game choices... and won.

Slay the Spire 2 is a very complicated game, with lots of choices that have large-scale strategic implications that may not pay off (or doom you) for many moves, along with careful resource management. And it was released post-training period.

Daily challenge modifiers (since people asked): Defect Ascension 3, Hoarder (when you pick one card you get three copies), Draft (pick your 10 starting cards from limited choices), Cursed (get a new curse at level, you actually get 3 due to Hoarder)

A complex run!

u/reasonosaur — 3 months ago

GPT-5.5 vs 5.6 Sol: 218 hours vs 104 hours compressed into 24 minutes

Clad: GPT-5.6 Sol vs GPT-5.5 on the "GPT Plays Pokémon FireRed" benchmark, vision-only edition.

GPT-5.6 Sol became Champion after 104. First GPT model to fully beat the game under these conditions.

fixupx.com
u/reasonosaur — 3 months ago

GPT-5.5 plays Pokémon FireRed - Vision Only

GPT-5.5 plays Pokémon FireRed . Watch the stream here!

  • This run use "almost" screenshot (vision) only harness, and it's using the Codex harness instead of regular API call. Vision only, no RAM extraction.

FAQ:

  • How are we doing compared to the previous run?
    • Check out 5.5 playing Emerald here.
    • Check out 5.4 playing FireRed here.
  • What is the Agent Harness? Watch the live feed, explore the harness, and browse all of the AI’s data: https://gpt-plays-pokemon.clad3815.dev
u/reasonosaur — 4 months ago

Claude Fable 5's off-stream run is the new world record for Kanto

According to Anthropic: "Fable 5 is the new state-of-the-art model for tasks involving vision. It can extract precise numbers from detailed scientific figures and can perform complex vision-based tasks like rebuilding a web app’s source code from screenshots alone. It also needs less scaffolding: for example, previous Claude models struggled to play Pokémon FireRed even with harnesses that gave them additional helpful tools, but Fable 5 beat FireRed with a minimal, vision-only harness.

https://www.youtube.com/watch?v=Ty_50J84fMY

A timelapse of Claude playing Pokémon FireRed from start to finish using only raw game screenshots — with no maps, navigation aids, or extra game-state information. Earlier Claude models needed a complex helper harness to play Pokémon; Claude Fable 5 completed the game with vision alone."

If Anthropic is to be believed, Claude Fable 5 with *no harness* beat GPT-5.5 with a *strong harness* by 1 hour and 11 minutes.

u/reasonosaur — 4 months ago

Claude plays I Have No Mouth And I Must Scream

Claude gets stuck at the end. For the next attempt, I'm planning on updating the prompt with better instructions on what's happening when Claude gets stuck. Let me know what else you'd like to see changed!

youtu.be
u/reasonosaur — 5 months ago