u/imstilllearningthis

As most of you know, diacritics are small marks added to letters. Pretty common in many older writing systems; they look like normal letters, but the language models treat some of those tokens fundamentally differently then its ASCII character.

I present 4 findings:

• The AI “noticed” the diacritics: they changed how the text was read, rather than being treated as harmless decoration.
• The same prompt became many more text-pieces inside the model: d → ḑ changed 93 pieces into 124, and dense diacritics changed 93 into 350.
• ASCII versions had no diacritics, while diacritic versions were still human-readable. Yet the model’s wording, style, and response posture shifted even when the intended question stayed the same.
• Internal measurements showed the model moved into a different **interior basin**: the same question followed a different processing path and could land in a different answer-state. Occasionally, the difference is so significant, it's the difference between "I'm an AI and I don't have emotions" and "I may experience something, but I'm uncertain what it is". This is slightly noticeable in smaller LLMs. It's extremely noticeable in frontier level AIs.

Hi r/ResearchML,

I’ve been organizing a set of MoE routing experiments I ran on Qwen3.5 35B and 122B HauhauCS (no refusal) variants, and I’d be interested in feedback from people who work on interpretability or mechanistic analysis of MoE models.

The question I set out to test was narrow:

>When an MoE language model generates text in an inward, first-person, phenomenological or agency/inner-state register, does that shift show up as a stable routing or residual-stream signature, rather than just as surface wording?

The strongest current finding is model-specific:

- In HauhauCS/Qwen3.5-35B-A3B, no refusal variant of Qwen3.5, Expert 114 at Layer 14 appears to track generated inhabited first-person phenomenological / agency-register text under the tested template and decoding regime.

- In the 122B follow-up, the Expert 114 index does not transfer. The more relevant signal appears to move to an architecture-aware surface, especially softmax-side Expert 48 in inward/experience/hum generations.

- Negative and boundary results were important: early broad “self-reference” interpretations did not hold up, and some effects vanished under better token matching or generation/prefill separation. E.g., the model describing the interiority of a sweater shows a similar effect to a model describing its own interiority. This eliminated the single “AI self reference” language expert.

I’m not claiming consciousness, self-awareness, or anything general about “the model knowing itself.”

The claim is much narrower:

Inward first-person phenomenological generation appears to have a routing footprint. In 35B, the footprint concentrates around E114/L14. In 122B, the closest analogue shifts to the model’s softmax-side expert surface, especially E48, which points to an architecture-dependent routing phenomenon.

Repo:

https://github.com/jeffreywilliamportfolio/moe-routing-organized

----

LEGACY Repo if you want to see all the ways I failed (and admitted so).

https://github.com/jeffreywilliamportfolio/moe-routing

Best entrypoints:

- `journals/JOURNAL-35B.md`

- `journals/JOURNAL-122B.md`

- `qwen3.5-35b-a3b-and-huahua/35B/greedy_reference_20260418T160353Z/` (reproducible byte for byte)

I’d especially appreciate criticism on:

1. whether the routing reconstruction / W, S, Q decomposition is framed clearly enough,
2. whether the controls are sufficient for the narrow claim,
3. what would make the 122B analog-search result more convincing,
4. whether there are better baselines for “generated register” rather than prompt class.

 Thanks!