AlphaZero · Visual notes
Teaching itself, one game at a time.
AlphaZero learns to play strong chess by playing against itself — no human games, no opening books, no hand-crafted evaluation.
Architecture · one iteration, animated
How the components wire together
One AZ iteration runs three phases in sequence:
self-play with best_net emits (s, π, z) tuples into the replay buffer,
training samples batches and updates candidate_net,
promotion copies the new weights back into best_net — then the loop restarts.
best_net, training updates candidate_net — never confuse the two.
(2) Phases are serial within an iteration (self-play → train → promote), but workers within self-play run in parallel.
(3) Promotion is unconditional (no arena gate); next iteration's self-play uses the just-promoted weights.
(4) Replay buffer is never reset between iterations.
Walk through one iteration
Four stages, in order.
Stage 1 · Self-play with MCTS
inside one MCTS simulation
MCTS: select → expand+evaluate → backup
For each move on the board, we run N simulations from the current state (50 for TTT, 800 for chess). Each simulation does three things, in order:
① Select — walk down the existing tree by picking, at each node, the child that maximizes PUCT:
a* = argmaxa [ Q(s, a) + cpuct · P(s, a) · √(Σb N(s, b)) / (1 + N(s, a)) ]② Expand & evaluate — on a fresh leaf, encode the state and run one forward pass of the NN to get priors P and a leaf value v ∈ [−1, +1]. Children are created for every legal action.
③ Backup — walk the path in reverse, flipping the sign of v at every ply (zero-sum: your opponent's good is your bad). At every node on the path:
N(s, a) ← N(s, a) + 1 W(s, a) ← W(s, a) + v_leaf (v flips sign at each ply) Q(s, a) ← W(s, a) / N(s, a)After all simulations, the visit distribution at the root is the MCTS-improved policy — sharper than the raw NN's prior because it integrates many moves ahead.
PUCT — the rule that picks each child
Exploit vs explore, per simulation
Not a reward function. PUCT — "Polynomial Upper Confidence applied to Trees" — is the rule that decides, at each step of an MCTS simulation, which child of the current node to descend into. For each candidate move a from node s:
PUCT(s, a) = Q(s, a) + c_puct · P(s, a) · √(Σ_b N(s, b)) / (1 + N(s, a)) └────────┘ └───────────────────────────────────────────┘ exploit: explore: average value scaled by NN prior P seen for a fades as N(s, a) grows
Q grows for promising moves as their average observed value rises. U shrinks for already-explored moves because N(s, a) appears in the denominator. So MCTS first explores broadly (guided by the NN prior P), then concentrates visits on whichever moves keep paying off.
c_puct is the only knob — typically 1.0–4.0. Higher = more exploration. The NN's prior P is what makes this polynomial upper confidence (instead of pure UCB1) — the NN biases exploration toward moves it already thinks are promising.
Q for every node on the path. PUCT then uses those updated Q's to guide the next simulation.
from search to training target
Visit distribution → πa via temperature τ
After N simulations at the root, every legal action a has a visit count N(s,a). We convert it to a policy:
πa ∝ N(s, a)1/τ
At τ = 1 the policy is proportional to visits — keeps stochasticity so games diverge.
In the limit τ → 0 the policy is argmax — committal play.
The switch happens at temperature_threshold (config field): plies with move_idx < threshold sample proportionally; later plies argmax.
closing the game out
Self-play completion: assign z per ply
When the game ends, the terminal value (+1 win, 0 draw, −1 loss) is stamped onto every ply from that ply's mover's point of view. Zero-sum: signs alternate along the ply track. The training tuple for each ply is:
(s, π, z) s : encoded canonical board at that ply π : MCTS visit policy at that ply (stage C) z : final outcome from that ply's mover's POVAll tuples from the game (with symmetries applied) get pushed into the replay buffer.
Stage 2 · Replay buffer
a FIFO queue of recent self-play tuples
Push in, sample out
A fixed-size FIFO queue holding training data from recent self-play games. Each ply of each self-play game emits one training tuple:
(s, π, z) s = encoded board state π = MCTS-improved policy at that ply (visit distribution) z = eventual game outcome from that player's perspective (+1 / 0 / -1)A TTT game produces ~5–9 tuples; a chess game produces ~80–150. The buffer keeps the most recent K tuples (50K for TTT, 1M for chess). Older ones age out as new self-play arrives. Each training step samples a random minibatch from the buffer.
Stage 3 · Training (ResNet + loss + AdamW)
the network being trained — a ResNet with two heads
Encoded state → ResNet body → two heads
The body is a residual network: stacked blocks where each block computes
y = ReLU(x + F(x)) with F(x) = conv → BN → ReLU → conv → BN. The + x skip is the entire trick — the derivative of y w.r.t. x equals 1 + ∂F/∂x, so gradients always reach early layers even if F's derivative is tiny.
"N blocks of K channels" sets the depth and width. For TTT: 4 × 32 is plenty. For chess (ours): 15 × 192; the original AlphaZero paper used 19 × 256 for chess and shogi.
The body's features split into two heads:
policy: logits z ∈ ℝᴬ → softmax: pa = exp(za) / Σb exp(zb) value : scalar z ∈ ℝ → tanh : v = tanh(z) ∈ [−1, +1]
The policy head emits raw logits; log_softmax is applied inside the loss (numerically stable). The value head squashes to [−1, +1] via tanh so it lives on the same scale as the game outcome z ∈ {−1, 0, +1}.
training step on one minibatch
The AlphaZero loss
Sample a minibatch (s, π, z) from the replay buffer. One forward pass produces (p, v) = fθ(s). The total loss has three pieces:
L(θ) = (z − v)² − π · log p + λ · ‖θ‖² └──────┘ └───────────┘ └─────────┘ value MSE policy CE L2 reg (over actions) (AdamW)
In the code: policy_loss = −(π · log_softmax(logits)).sum(−1).mean() and
value_loss = mse_loss(v, z). log_softmax is fused for numerical stability. The L2 term comes for free from AdamW(weight_decay=λ) — it's not added to the loss expression.
the parameter update
AdamW step
mt = β1 mt−1 + (1−β1) gt vt = β2 vt−1 + (1−β2) gt² m̂t = mt / (1 − β1t) v̂t = vt / (1 − β2t) θ ← θ − η · ( m̂t / (√v̂t + ε) + λ · θ )Adam = adaptive per-parameter step from first/second gradient moments. AdamW = decoupled weight decay (λ · θ added directly to the update, not via the loss). Hundreds of these steps run per iteration.
Stage 4 · Promotion
closing the iteration
Promote — unconditional
best_net.load_state_dict(candidate_net.state_dict())
The trained candidate_net becomes the new best_net. Next iteration's self-play uses it. No head-to-head match, no acceptance threshold — matches the AlphaZero (2017) paper.
Reference
Activation functions, at a glance
The four nonlinearities (and one normalizer) that the AlphaZero net actually uses, and where each one lives.
ReLU
Every conv block (input projection, residual blocks, both head projections, value FC hidden).
BatchNorm
After every conv (input + each residual + each head's 1×1). Stabilizes training; per-channel μ, σ.
Softmax
Policy head, inside the loss (log_softmax for stability). Turns logits into a probability over actions.
tanh
Final layer of the value head only. Matches the game outcome scale z ∈ {−1, 0, +1}.
Channels vs blocks — what each one buys you.
"15 blocks × 192 channels" is the chess net we train. What do those two numbers actually mean, and how would you change them?
1 · What is a channel?
Every square of the 8×8 board carries a vector of C numbers; those C numbers are the C channels at that location. Channels are not spatial — they're the feature dimension layered on top of position.
Input (encoded chess position): 20 channels = 12 piece planes (6 white + 6 black) + 4 castling + 1 side-to-move + 1 en-passant + 2 move clocks.
After input conv: projected up to 192 channels — the network's working "width."
2 · Channels vs blocks
Channels — vocabulary per square
Per-square expressiveness. At one square, the net can encode "is this an attacked rook on a half-open file under pin pressure" only if there's enough channels to dedicate dimensions to those concepts.
Cost: compute & params scale as channels² (the 3×3 conv has C inputs and C outputs).
Blocks — editing passes & reach
Reach + abstraction. A single 3×3 conv only sees neighbors. You need ≥ 7 blocks for one square to "see" the whole 8×8 board, plus extra blocks to combine what it sees.
Cost: compute & params scale linearly with blocks.
3 · How to size them
Three constraints push back on you. Pick the smallest net that satisfies all three.
| Constraint | Formula | For our 15 × 192 |
|---|---|---|
| Compute per forward | O(blocks · channels²) |
~50 ms on Mac MPS |
| Receptive field ≥ board | need ≥ 7–8 blocks for 8×8 |
15 blocks → fully seen ~7× |
| Params vs data | corpus / params > ~5 |
3M / 10.6M = 0.3 → data-limited |
Param count rule of thumb:
params ≈ 18 · blocks · channels².
For 15 × 192: 18 × 15 × 192² ≈ 9.96M — matches the 10.6M we measure.
| Variant | Blocks × Channels | Params |
|---|---|---|
| AZ paper (chess / shogi) | 20 × 256 | ~23M |
| Mini-AZ (TTT / Connect4) | 4–8 × 64–96 | 0.5–1M |
| Ours (chess on Mac) | 15 × 192 | 10.6M |
- Pick channels first — sets feature richness. 128 / 192 / 256 are the common sizes.
- Pick blocks so receptive field ≥ board size, plus 2–4 extra for abstraction passes.
- Sanity-check params vs data — corpus should comfortably exceed param count.
- Sanity-check forward time — if too slow, cut channels before cutting blocks: compute scales as channels² but only linearly with blocks.
Reference
Glossary
Terms that appear repeatedly. Grouped by area. Use this as a quick lookup while reading code or the design doc.
Core RL & game concepts
The full board position at a point in the game. For TTT: a 3×3 array; for chess: piece placement + castling rights + en-passant + counters.
A legal move from a state. Represented as an integer index into the action space (0–8 for TTT, ~4672 for chess).
A probability distribution over actions. Two flavors: the NN's prior P, and the MCTS-improved policy π used as a training target.
Expected game outcome in [−1, +1]. v is the NN's predicted value; z is the actual realized outcome of the game.
Agent plays against itself to generate training data. Each ply produces one (s, π, z) tuple. Source of all training signal.
One full cycle: self-play games → buffer → training steps → promote → (loop). One iteration ≡ one "epoch" of AZ training.
One complete self-play game from initial_state to a terminal position. Produces a batch of training tuples sharing the same outcome z.
Board rewritten from the current player's POV so the NN always "plays as +1." Halves the effective state space the NN must learn.
Game-equivalent transformations (rotations, reflections) used for data augmentation. TTT has 8; Connect 4 has 2; chess has none.
MCTS terms
The NN's pre-search probability for each move at a node. Set once when the node is expanded; never updated by search.
How many MCTS simulations have descended through a node. The final visit distribution at the root is the MCTS-improved policy π.
Average value seen for an action so far: Q = W / N where W is the summed value. The "exploit" half of PUCT.
c_puct · P · √(ΣN) / (1+N). Decays as the action's N grows. The "explore" half of PUCT.
Exploration constant. Higher = more exploration. Typically 1.0–4.0. We use 1.5 for TTT.
Random perturbation of root priors during self-play: P' = 0.75·P + 0.25·Dir(α). Ensures diverse openings. α=1.0 for TTT.
Controls how the visit distribution is converted to an action. τ=1: sample proportional to visits. τ=0: argmax (greedy).
Propagating a leaf's value up the search path, with sign flips per ply (zero-sum). Updates N, W (and so Q) along the path.
The function MCTS calls at every leaf: (state) → (priors, value). In practice it's the NN, but MCTS doesn't know that.
Neural network & training
The main body of the NN — stacked residual blocks that produce a shared feature representation consumed by both heads.
A small final sub-network on top of the backbone. AlphaZero has two: policy head (action logits) and value head (scalar in [−1,+1]).
y = ReLU(x + F(x)) where F is two convs + BatchNorm + ReLU. The +x is the skip connection.
The +x path that lets gradients flow through deep networks. Makes the derivative ∂y/∂x = 1 + ∂F/∂x — the "1" prevents vanishing gradients.
One feature map produced by a conv layer. "32 channels" = 32 different feature maps per spatial position. More channels = wider network.
The weights of one convolution — a small grid (3×3 typically) slid across the input to produce one channel of output.
Normalizes activations across the batch to stabilize training. Sits between conv and activation in residual blocks.
Used for the policy head. Measures distance between predicted distribution and target π: −Σ π · log(softmax(logits)).
Mean squared error. Used for the value head: (v − z)² averaged across the batch.
Penalizes large weights to prevent overfitting. Applied via AdamW(weight_decay=1e-4).
Optimizer = Adam + decoupled weight decay. The modern default; converges faster than plain SGD on small games. The paper used SGD+momentum.
FIFO queue of recent training tuples. Capacity 50K for TTT, 1M for chess. Uniform random sampling — no prioritization.
Two copies of the NN. best_net is the frozen network used for self-play. candidate_net is the one being trained; it is unconditionally promoted to best every iteration.
One minibatch: sample from buffer → forward → loss → backward → optimizer.step. Hundreds happen per iteration.