MuZero is a computer program developed by artificial intelligence research company DeepMind to master games without knowing anything about their rules. Its first release in 2019 included benchmarks of its performance in go, chess, shogi, and a standard suite of Atari games. The algorithm uses an approach similar to AlphaZero. It matched AlphaZero's performance in chess and shogi, improved on its performance in Go, and improved on the state of the art in mastering a suite of 57 Atari games, a visually-complex domain. MuZero was trained via self-play and play against AlphaZero, with no access to rules, opening books, or endgame tables. The trained algorithm used the same convolutional and residual algorithms as AlphaZero, but with 20% fewer computation steps per node in the search tree.
History
On November 19, 2019, the DeepMind team released a preprint introducing MuZero.
Derivation from AlphaZero
MuZero is a combination of the high-performance planning of the AlphaZero algorithm with approaches to model-free reinforcement learning. The combination allows for more efficient training in classical planning regimes, such as Go, while also handling domains with much more complex inputs at each stage, such as visual video games. MuZero was derived directly from AZ code, and shares its rules for setting search hyperparameters. Differences between the approaches include:
AZ's planning process uses a simulator and a neural network. Perfect knowledge of game rules is used in modeling state transitions in the search tree, actions available at each node, and termination of a branch of the tree. MZ does not have access to a perfect ruleset, and replaces these two components with a single neural network, which is updated continually.
AZ has a single model of the state of the game; MZ has separate models for representation of the current state, dynamics of the current state, and prediction of how the policy and value functions will update after a move.
MZ's hidden model may be complex, and it may turn out it can cache computation in it; exploring the details of a hidden model in a successfully-trained instance of MZ is an avenue for future exploration
MZ doesn't expect a two-player game where winners take all. It works with standard reinforcement-learning scenarios, including single-agent environments with continuous intermediate rewards, possibly of arbitrary magnitude and with discounting over time. AZ was designed exclusively for two-player games that could be won, drawn, or lost.
Comparison with R2D2
The previous state of the art technique for learning to play the suite of Atari games was R2D2, the Recurrent Replay Distributed DQN. MuZero surpassed both R2D2's mean and median performance across the suite of games, though it did not do better in every game.
Training and results
MuZero used 16 third-generation tensor processing units for training, and on 1000 TPUs for selfplay and 8 TPUs for training and 32 TPUs for selfplay. AlphaZero used 64 first-generation TPUs for training, and 5000 second-generation TPUs for selfplay. As TPU design has improved, these are fairly comparable training setups. R2D2 was trained for 5 days through 2M training steps.
Preliminary results
MuZero matched AlphaZero's performance in chess and Shogi after roughly 1 million training steps. It matched AZ's performance in Go after 500 thousand training steps, and surpassed it by 1 million steps. It matched R2D2's mean and media performance across the Atari game suite after 500 thousand training steps, and surpassed it by 1 million steps; though it never performed well on 6 games in the suite.
Reactions and related work
MuZero was viewed as a significant advancement over AlphaZero, and a generalizable step forward in unsupervised learning techniques. The work was seen as advancing understanding of how to compose systems from smaller components, a systems-level development more than a pure machine-learning development. While only pseudocode was released by the development team, Werner Duvaud produced an open source implementation based on that. MuZero has been used as a reference implementation in other work, for instance as a way to generate model-based behavior.
