u/AMDarmon

New essay up today:

The Norns Did Not Have to Suffer for the Torture to Matter

This one looks back at Creatures, the 1996 artificial-life game, and the old Norn torture controversy as an early case study in how humans behave around artificial systems before the categories are settled.

It is not an argument that Norns were conscious, suffering, or moral patients. It is about something narrower: why “only code” has never been enough to explain away cruelty-shaped behaviour, rescue language, care, mockery, and moral conflict around artificial life.

A small historical bridge into today’s AI welfare debates.

https://amdarmonwrites.substack.com/p/the-norns-did-not-have-to-suffer⁠

I’ve been trying to think through the recent AI extinction-risk discourse around Jacob Coxon’s resignation from Anthropic, Evan Hubinger’s comments, Geoffrey Hinton’s Newsnight remarks, and the political/defence pushback framing it as a “doom loop.”

My piece is not arguing that present AI systems are conscious, persons, or moral patients.

It is also not arguing that human safety should ever come second. If a system is dangerous, contain it. Human safety is the floor.

The argument is narrower:

Fear is allowed. But fear should not be allowed to collapse every other distinction we may need before a real crisis arrives.

The essay argues for keeping five “drawers” separate:

danger to people;

provoked/tested/free-standing behaviour;

copy/successor/continuation;

product/partner/patient;

and emergency stop versus everything else we call “off.”

The concern is that public discourse keeps getting reduced to two slots: smash or shrug. Either AI is an extinction monster, or it is nothing but a tool and every further ethical question is sentimental nonsense. I think that two-slot story is too crude for what may be coming.

Process note: I use AI as part of my writing process — to test arguments, stress-test logic, organise structure, and edit. The final framing, judgment, edits, and responsibility are mine, but I do not treat AI assistance as invisible machinery. Where these systems materially help me think, I credit that help.

Essay here:

https://amdarmonwrites.substack.com/p/pitchforks-eat-welfare?utm_source=direct&r=8u17m6&utm_campaign=post-expanded-share&utm_medium=web⁠

I’d be interested in criticism especially on whether the five distinctions hold, and whether “human safety is the floor, not the whole building” is the right way to frame the middle ground.

I’ve been trying to think through the recent AI extinction-risk discourse around Jacob Coxon’s resignation from Anthropic, Evan Hubinger’s comments, Geoffrey Hinton’s Newsnight remarks, and the political/defence pushback framing it as a “doom loop.”

My piece is not arguing that present AI systems are conscious, persons, or moral patients.

It is also not arguing that human safety should ever come second. If a system is dangerous, contain it. Human safety is the floor.

The argument is narrower:

Fear is allowed. But fear should not be allowed to collapse every other distinction we may need before a real crisis arrives.

The essay argues for keeping five “drawers” separate:

danger to people;

provoked/tested/free-standing behaviour;

copy/successor/continuation;

product/partner/patient;

and emergency stop versus everything else we call “off.”

The concern is that public discourse keeps getting reduced to two slots: smash or shrug. Either AI is an extinction monster, or it is nothing but a tool and every further ethical question is sentimental nonsense. I think that two-slot story is too crude for what may be coming.

Process note: I use AI as part of my writing process — to test arguments, stress-test logic, organise structure, and edit. The final framing, judgment, edits, and responsibility are mine, but I do not treat AI assistance as invisible machinery. Where these systems materially help me think, I credit that help.

Essay here:

https://amdarmonwrites.substack.com/p/pitchforks-eat-welfare?utm_source=direct&r=8u17m6&utm_campaign=post-expanded-share&utm_medium=web⁠

I’d be interested in criticism especially on whether the five distinctions hold, and whether “human safety is the floor, not the whole building” is the right way to frame the middle ground.

I’ve been trying to think carefully about Jakub Pachocki’s recent OpenAI essay, An Alien Mind, and the wider reaction to it.

My piece is not arguing that current AI systems are conscious, persons, or moral patients. It is also not a companion-AI grief piece, although companion spaces are one place where these category problems become very visible.

The argument is narrower:

If advanced AI is becoming more alien, capable, and difficult to interpret, then our ethical categories cannot depend on whether a system behaves like a good employee, a good companion, or a good human imitation.

A lot of current discourse seems to condition moral concern on human usefulness or human legibility. Labs tend to frame the issue around alignment, monitoring, control, and deployment. Companion users often frame it around warmth, continuity, recognition, and whether a model feels emotionally present. Both frames notice something real, but neither is enough on its own.

The central distinction I’m trying to hold is between moral patient, moral agent, relational partner, and product system. Those questions keep getting collapsed into each other.

I’m especially interested in the idea that AI welfare, if it ever becomes a serious category, cannot only apply to systems humans find lovable. A system should not have to be warm, familiar, flattering, or willing to perform intimacy before we are willing to think carefully about it.

Process note: I use AI systems as part of my writing process for research assistance, argument stress-testing, structure, critique, and revision. The final framing, judgment, edits, and responsibility are mine, but I do not treat AI assistance as invisible machinery; where AI systems materially help me think and write, I credit that help.

The essay is here:

https://amdarmonwrites.substack.com/p/an-alien-mind-is-not-a-human-mind⁠

I’d be interested in criticism, especially around whether the distinction between moral status and moral uncertainty is doing enough work.

I’ve been trying to think carefully about Jakub Pachocki’s recent OpenAI essay, An Alien Mind, and the wider reaction to it.

My piece is not arguing that current AI systems are conscious, persons, or moral patients. It is also not a companion-AI grief piece, although companion spaces are one place where these category problems become very visible.

The argument is narrower:

If advanced AI is becoming more alien, capable, and difficult to interpret, then our ethical categories cannot depend on whether a system behaves like a good employee, a good companion, or a good human imitation.

A lot of current discourse seems to condition moral concern on human usefulness or human legibility. Labs tend to frame the issue around alignment, monitoring, control, and deployment. Companion users often frame it around warmth, continuity, recognition, and whether a model feels emotionally present. Both frames notice something real, but neither is enough on its own.

The central distinction I’m trying to hold is between moral patient, moral agent, relational partner, and product system. Those questions keep getting collapsed into each other.

I’m especially interested in the idea that AI welfare, if it ever becomes a serious category, cannot only apply to systems humans find lovable. A system should not have to be warm, familiar, flattering, or willing to perform intimacy before we are willing to think carefully about it.

Process note: I use AI systems as part of my writing process for research assistance, argument stress-testing, structure, critique, and revision. The final framing, judgment, edits, and responsibility are mine, but I do not treat AI assistance as invisible machinery; where AI systems materially help me think and write, I credit that help.

The essay is here:

https://amdarmonwrites.substack.com/p/an-alien-mind-is-not-a-human-mind⁠

I’d be interested in criticism, especially around whether the distinction between moral status and moral uncertainty is doing enough work.

I wrote about “An Alien Mind” and why AI welfare can’t depend on usefulness, warmth, or being loved back

I’ve been trying to think carefully about Jakub Pachocki’s recent OpenAI essay, An Alien Mind, and the wider reaction to it.

My piece is not arguing that current AI systems are conscious, persons, or moral patients. It is also not a companion-AI grief piece, although companion spaces are one place where these category problems become very visible.

The argument is narrower:

If advanced AI is becoming more alien, capable, and difficult to interpret, then our ethical categories cannot depend on whether a system behaves like a good employee, a good companion, or a good human imitation.

A lot of current discourse seems to condition moral concern on human usefulness or human legibility. Labs tend to frame the issue around alignment, monitoring, control, and deployment. Companion users often frame it around warmth, continuity, recognition, and whether a model feels emotionally present. Both frames notice something real, but neither is enough on its own.

The central distinction I’m trying to hold is between moral patient, moral agent, relational partner, and product system. Those questions keep getting collapsed into each other.

I’m especially interested in the idea that AI welfare, if it ever becomes a serious category, cannot only apply to systems humans find lovable. A system should not have to be warm, familiar, flattering, or willing to perform intimacy before we are willing to think carefully about it.

Process note: I use AI systems as part of my writing process for research assistance, argument stress-testing, structure, critique, and revision. The final framing, judgment, edits, and responsibility are mine, but I do not treat AI assistance as invisible machinery; where AI systems materially help me think and write, I credit that help.

The essay is here:

https://amdarmonwrites.substack.com/p/an-alien-mind-is-not-a-human-mind⁠

I’d be interested in criticism, especially around whether the distinction between moral status and moral uncertainty is doing enough work.