Mini-LLM — every computation step

Sentence to generate the next token from:

Computed so far — click a card to jump to that step

Residual stream — token_vectors at this point of the computation (one row per token of the sentence)

Complete run: next-token generation

Training data & vocabulary

The complete knowledge of the model. Everything it ever predicts comes from the regularities of these sentences; the vocabulary is derived from them, not maintained by hand. Click a word to see its embedding row — the numbers looked up in step 1 — and the sentences it was learned from.

Training sentences

    Vocabulary

    Weights — the learned parameters

    Code — training & inference

    The three files that produced everything on this page: train.py learns the weights, inference.py generates text with them, model.py contains the math both call. The listings are the files themselves, read when the page was built — not a version rewritten for the presentation.