u/EndOpening7942

One explanation that seems to help beginners is to stop starting with "the transformer" and instead follow one token through the machine.

My current mental model:

1. Text is split into tokens.
2. Each token becomes an embedding vector.
3. That vector becomes a hidden state: the model's current internal version of the token.
4. Each layer rewrites the hidden state using context.
5. Attention is the "which earlier tokens matter right now?" mechanism.
6. Feed-forward / expert layers transform the representation after context has been mixed in.
7. The final hidden state is projected into logits over the vocabulary.
8. Softmax/sampling turns those logits into the next token.

The key simplification is that the model is not "thinking in words." It is repeatedly rewriting vectors until the last vector is useful enough to predict what comes next.

For learners, I think this ordering is less intimidating than jumping straight into Q/K/V matrices:

tokens -> embeddings -> hidden states -> context mixing -> logits -> next token

Curious how others here explain hidden states or attention to beginners. What analogy has worked best for you?

One explanation that seems to help beginners is to stop starting with "the transformer" and instead follow one token through the machine.

My current mental model:

1. Text is split into tokens.
2. Each token becomes an embedding vector.
3. That vector becomes a hidden state: the model's current internal version of the token.
4. Each layer rewrites the hidden state using context.
5. Attention is the "which earlier tokens matter right now?" mechanism.
6. Feed-forward / expert layers transform the representation after context has been mixed in.
7. The final hidden state is projected into logits over the vocabulary.
8. Softmax/sampling turns those logits into the next token.

The key simplification is that the model is not "thinking in words." It is repeatedly rewriting vectors until the last vector is useful enough to predict what comes next.

For learners, I think this ordering is less intimidating than jumping straight into Q/K/V matrices:

tokens -> embeddings -> hidden states -> context mixing -> logits -> next token

Curious how others here explain hidden states or attention to beginners. What analogy has worked best for you?