Connect 4 on the AlphaZero Framework (Sub-project 2)

date:2026-05-15 status:Approved (brainstorming → implementation) sub project:2 target:Validate the framework on Connect 4 — beat minimax-depth-8 in ≥80% of 200 games

1. Context

Sub-project 1 built and validated the game-agnostic AlphaZero framework on Tic-Tac-Toe. Sub-project 2's job is to demonstrate that the framework scales to a meaningfully harder game without changing any framework code — only a new Game subclass and a new evaluation opponent.

Connect 4 is the ideal stepping stone between TTT and chess:

Property TTT Connect 4 Chess
Board 3×3 = 9 cells 6×7 = 42 cells 8×8 = 64 cells
Action space 9 7 ~4672
Reachable states ~5,478 ~10¹³ ~10⁴³
Max game length 9 plies 42 plies unbounded (we cap)
Branching factor ~9 → 1 ~7 throughout ~35 avg
Solved? Yes (draw) Yes (1st player wins) No

Connect 4 is 10⁹× larger than TTT in state space but still tractable for from-scratch self-play training on a consumer machine. If the framework works on Connect 4, it'll work on chess (modulo compute scale).

2. Goals

3. Non-goals

4. What's new vs what's reused

The architecture is unchanged. Only three new source files and three new tests.

Component Status What changes
Game ABC reuse as-is nothing
TicTacToe leave in place nothing
Connect4 new rules, encoding, terminal detection, symmetries
AlphaZeroNet reuse as-is only input_shape and action_size differ (parameterized by Game)
MCTS reuse as-is nothing
SelfPlayWorker reuse as-is nothing
ReplayBuffer reuse as-is nothing
Arena reuse as-is nothing
Connect4MinimaxOpponent new α-β pruning at depth 8 + heuristic eval
Trainer small extension replace TTT-specific _eval_vs_solver with a generic _eval_vs_opponent(opponent) callable injection
cli.py small generalization train/eval/play dispatch on --game {tictactoe, connect4}

This is the payoff of the layered architecture in Sub-project 1.

5. Components

5.1 Connect4 (new Game subclass)

State representation: np.ndarray of shape (6, 7), dtype int8, values +1 / −1 / 0.

Action space: integers 0..6 corresponding to the 7 columns.

Interface:

class Connect4(Game):
    @property
    def input_shape(self) -> tuple[int, int, int]:
        return (3, 6, 7)              # (channels, H, W) — my, opp, ones planes

    @property
    def action_size(self) -> int:
        return 7

    def initial_state(self) -> State:
        return np.zeros((6, 7), dtype=np.int8)

    def current_player(self, state: State) -> int:
        moves = int(np.count_nonzero(state))
        return 1 if moves % 2 == 0 else -1

    def legal_actions_mask(self, state: State) -> np.ndarray:
        # A column is legal iff its top cell (row 0) is empty.
        return state[0, :] == 0

    def apply(self, state: State, action: int) -> State:
        # Drop piece into lowest empty cell of column `action`.
        # Raise ValueError if column is full.
        ...

    def terminal_value(self, state: State) -> float | None:
        # Check 4-in-a-row in 4 directions: horizontal, vertical, both diagonals.
        # Return -1.0 (opponent of current player just won), 0.0 (draw), or None (in progress).
        ...

    def encode(self, state: State) -> np.ndarray:
        # 3 planes: my pieces, opp pieces, ones. Assumes canonical state.
        ...

    def canonical_state(self, state: State) -> State:
        return (state * self.current_player(state)).astype(np.int8)

    def symmetries(
        self, encoded: np.ndarray, policy: np.ndarray
    ) -> list[tuple[np.ndarray, np.ndarray]]:
        # Identity + horizontal mirror. The mirror flips columns left↔right
        # in both the encoded board planes and the policy.
        identity = (encoded, policy)
        mirrored = (
            np.flip(encoded, axis=2).copy(),
            np.flip(policy, axis=0).copy(),
        )
        return [identity, mirrored]

Key decisions:

5.2 Connect4MinimaxOpponent (new agent)

Standard α-β minimax with a position-evaluation heuristic for non-terminal leaves.

Interface:

class Connect4MinimaxOpponent:
    """Callable agent: takes (game, state) → action. Uses α-β minimax to depth N."""

    def __init__(self, depth: int = 8):
        self.depth = depth

    def __call__(self, game: Game, state: State) -> int:
        # Search with α-β pruning to self.depth; return best action.
        ...

    def _evaluate(self, state: State) -> float:
        """Static evaluation of a non-terminal position from current player's POV.

        Score = sum over 4-cell windows of:
            +∞  if window contains 4 of mine
            -∞  if window contains 4 of opponent
            +w[k]  if window contains k of mine + (4-k) empties (no opp)
            -w[k]  if window contains k of opp + (4-k) empties (no mine)
            0   otherwise (window has both)

        Plus a small center-bias bonus: each piece in the center column adds
        a constant for the owning side.
        """
        ...

Heuristic weights (starting values, may tune for opponent strength):

Window content Weight
3 of mine + 1 empty +5
2 of mine + 2 empties +2
Center column piece +3 each
Symmetric for opponent mirrored negative

Why this works: At depth 8 with α-β pruning over a 7-action branching factor, the search visits roughly 7⁴-7⁵ ≈ 16K-100K nodes per move (after pruning). Pure Python with numpy primitives evaluates ~1-2ms per leaf, giving ~10-200ms per move — fast enough for 200-game evaluation matches that finish in minutes.

Test target: Depth-8 should beat depth-2 in ≥90% of games (confirms the search is correctly exploring).

5.3 Generalizing the Trainer's eval hook

Current Sub-project 1 code has a hardcoded _eval_vs_solver that imports solve_tictactoe_action. To support Connect 4 cleanly, we generalize to:

def _eval_vs_opponent(self, opponent: Agent, num_games: int) -> dict:
    """Play current best_net vs an arbitrary opponent. Game-agnostic."""
    from .arena import play_match
    net_agent = self._make_argmax_mcts_agent(self.best_net)
    result = play_match(self.game, net_agent, opponent, num_games=num_games)
    return {"wins": result.wins_a, "draws": result.draws, "losses": result.losses_a}

Trainer.run() then takes an optional eval_opponent: Agent | None parameter. For TTT, the existing solver_agent wrapper is passed; for Connect 4, a Connect4MinimaxOpponent(depth=8) instance. The old _eval_vs_solver becomes a thin wrapper around _eval_vs_opponent to preserve backward compatibility with the Sub-project 1 E2E test.

5.4 Minor CLI generalization

cli.py currently hardcodes game = TicTacToe(). Extend to:

GAMES = {"tictactoe": TicTacToe, "connect4": Connect4}

# In each subcommand:
game_cls = GAMES[args.game]
game = game_cls()

Add --game {tictactoe,connect4} to train, eval, play subcommands; default tictactoe to preserve Sub-project 1 behavior. For eval --game connect4, the opponent is Connect4MinimaxOpponent(depth=args.opponent_depth or 8).

6. Hyperparameters (starting config)

configs/connect4.toml:

# Connect 4 training config — starting values from the AlphaZero paper
# scaled down for a learning project on M4 Pro.

# Network
n_blocks = 6                       # up from TTT's 4 — deeper for 6×7 board
n_channels = 64                    # up from TTT's 32
                                   # ≈ 600K params (vs TTT's 76K)

# MCTS
num_simulations = 100              # up from TTT's 50
c_puct = 2.5                       # up from TTT's 1.5 — more exploration with 7-way branching
dirichlet_alpha = 0.5              # down from TTT's 1.0 — concentrated noise on fewer actions
dirichlet_weight = 0.25

# Self-play
games_per_iteration = 100
temperature_threshold = 15         # up from TTT's 6 — Connect 4 games ~20-30 plies

# Training
training_steps_per_iteration = 1000  # up from TTT's 500
batch_size = 64
learning_rate = 1e-3
weight_decay = 1e-4

# Replay buffer
replay_buffer_capacity = 200_000   # up from 50K — longer games, more iterations
min_buffer_size = 20_000

# (no arena gate — Trainer always promotes candidate to best;
# see "Lessons from Sub-project 1" below.)

# Eval (vs minimax-depth-8)
eval_interval = 5
eval_games = 200

# Schedule
num_iterations = 50

# Misc
seed = 42
device = "auto"
checkpoint_dir = "checkpoints"
log_dir = "runs"

Wall-time estimates on M4 Pro:

Setup Per iter 50 iter total
CPU ~15-25 min 12-20 hours
MPS ~5-10 min 3-6 hours

If too slow, the first knob to drop is games_per_iteration (50 instead of 100) — halves training time at modest quality cost.

The handful of decisions that actually matter:

Lessons from Sub-project 1 (arena gating)

Sub-project 1's plan followed AlphaGo Zero (2017) and gated network promotion behind win_rate ≥ 0.55 over 40 candidate-vs-best games. This completely stalled TTT training in practice:

  1. After the first acceptance (~iter 5), candidate and best are very similar networks. Wrapped in MCTS at 50 simulations per move, both produced similar argmax actions → most games drew.
  2. win_rate = (wins + 0.5·draws) / total therefore hovered near 0.50 — never reaching 0.55.
  3. Every iteration: candidate trained → arena rejected → candidate weights reverted to best's. Identical self-play data generated next iter. Infinite loop.
  4. Result after 46 iterations on MPS: best_net weights were byte-identical to their iter 5 state. The frozen network had learned a defeatist O policy (prior on center after X's corner opening = 0.095, below uniform 1/9), and lost 100% of O-side games against a perfect solver.

Fix: removed the arena gate from Trainer.run() entirely. _maybe_accept_candidate was replaced by _promote_candidate_to_best, which is called unconditionally after every training iteration — matching the AlphaZero paper (2017), which deliberately dropped the arena step that AlphaGo Zero had used. Training then accumulated improvements iteration-over-iteration, and the same framework reached zero losses against the perfect TTT solver at iteration 10.

The _arena_win_rate helper is preserved as an optional diagnostic — useful for inspecting "is the candidate stronger than best?" without gating on the answer. We may use it for logging during Connect 4 training, just not for gating.

For Sub-project 2, we adopt the AlphaZero convention from the start: always promote, no gate. We may revisit arena-style safeguards in Sub-project 3 (chess) if training instability requires a safety net — but the gating threshold value (if used at all) will need to be far below 0.55 (the AlphaZero paper experiments suggest 0.50–0.52 in their ablations).

7. Testing strategy

7.1 Connect4 game tests (tests/test_game_connect4.py)

About 20 tests, mirroring tests/test_game_tictactoe.py:

7.2 Connect4MinimaxOpponent tests (tests/test_connect4_minimax.py)

7.3 Kill-criterion E2E (tests/test_e2e_connect4.py, @pytest.mark.slow)

@pytest.mark.slow
def test_e2e_connect4_beats_minimax_depth_8(tmp_path, monkeypatch):
    monkeypatch.chdir(tmp_path)
    config = load_config(REPO_ROOT / "configs/connect4.toml")
    trainer = Trainer(Connect4(), config)
    trainer.run(eval_opponent=Connect4MinimaxOpponent(depth=8))

    agent = trainer._make_argmax_mcts_agent(trainer.best_net)
    opponent = Connect4MinimaxOpponent(depth=8)
    result = play_match(Connect4(), agent, opponent, num_games=200)

    print(f"\nFinal: wins={result.wins_a} draws={result.draws} losses={result.losses_a} "
          f"win_rate={result.win_rate:.2%}")
    assert result.win_rate >= 0.80, (
        f"Agent win rate {result.win_rate:.2%} below 0.80 target. "
        "Increase num_iterations or num_simulations."
    )

7.4 Invariant tests

The existing tests/test_invariants.py already guards the 6 framework invariants. No duplication needed — those tests operate on TicTacToe via fixtures, but they prove the framework's invariants hold, which is what matters. If we want, we can later parameterize them across (TTT, C4) for extra safety.

8. Success criterion (recap)

Sub-project 2 is done when:

pytest -m slow tests/test_e2e_connect4.py passes consistently with result.win_rate >= 0.80 against the depth-8 minimax opponent over 200 games (alternating colors).

After this passes, Sub-project 3 (chess) can begin.

9. Project layout additions

configs/connect4.toml                              ← new
src/alphazero/games/connect4.py                    ← new (~200 lines)
src/alphazero/opponents/__init__.py                ← new package
src/alphazero/opponents/connect4_minimax.py       ← new (~200 lines)
src/alphazero/trainer.py                           ← small change: generalize _eval_vs_*
src/alphazero/cli.py                               ← small change: --game flag
tests/test_game_connect4.py                        ← new (~250 lines)
tests/test_connect4_minimax.py                     ← new (~150 lines)
tests/test_e2e_connect4.py                         ← new (~50 lines)

The new opponents/ package is parallel to the existing solvers/. solvers/ is for ground-truth oracles (perfect TTT minimax); opponents/ is for classes of agents used as eval baselines. Same conceptual layer; different semantic role.

10. Out of scope (saved for future sub-projects)

11. References