Karpathy: Building GPT

My starting point was Karpathy's tiny Shakespeare example: a small model I could train and use to generate text. This track connects micrograd's explanation of backpropagation to that exercise and the larger GPT-2 implementation.

Micrograd: Backpropagation from Scratch🔗

References: micrograd code · The spelled-out intro to neural networks and backpropagation.

Micrograd makes backpropagation visible by building a small automatic differentiation engine around scalar values. Each operation records its inputs and how to pass a gradient back to them.

Forward pass: calculate values and build the computation graph.
Chain rule: multiply the gradient arriving from later operations by the local derivative.
Multiple paths: add gradient contributions when a value influences the result through more than one path; this is why the implementation uses +=.
Backward pass: visit nodes in reverse topological order so downstream contributions arrive before a node propagates its gradient further.

For one path through an intermediate value z:

\frac{\partial L}{\partial x} = \frac{\partial L}{\partial z}\,\frac{\partial z}{\partial x}

The Value engine explains what loss.backward() computes. Updating the weights using those gradients is a separate optimizer step.

Tiny Shakespeare🔗

Reference: Karpathy's nanoGPT video lecture code.

Dataset: tiny Shakespeare — raw text: input.txt.

The small example makes the full training cycle easier to follow: encode text, build input/next-token target batches, compute predictions and loss, update weights, then sample new text one token at a time.

GPT-2🔗

Reference: Karpathy's build-nanogpt walkthrough, which builds toward reproducing GPT-2 (124M).

This is the next scale of the same learning track: understanding how the model, tokenizer, data pipeline, and training loop fit together. For the compute accounting, see why training costs approximately 6ND FLOPs.