<p>Yes!</p>
<p><a href="https://github.com/LeelaChessZero/lc0" rel="nofollow noreferrer">LeelaChessZero</a>, an open-source re-implementation and continuation of AlphaZero, has been experimenting with this for a while now. Their strongest networks are currently transformers, not convolution-based ResNets anymore.</p>
<p>Some resources:</p>
<ul>
<li>The <a href="https://lczero.org/blog/2023/01/2022-progress/#network-architecture" rel="nofollow noreferrer">2022 progress</a> blog post</li>
<li>More details on the network architecture as of 7 months ago <a href="https://github.com/Ergodice/lczero-training/blob/attention-net-body/README.md" rel="nofollow noreferrer">here</a></li>
<li>A <a href="https://lczero.org/blog/2024/02/how-well-do-lc0-networks-compare-to-the-greatest-transformer-network-from-deepmind/" rel="nofollow noreferrer">recent comparison</a> to the latest DeepMind paper, which also uses transformers.</li>
<li>A <a href="https://lczero.org/blog/2024/02/transformer-progress/" rel="nofollow noreferrer">recent blog post</a> about the current best transformer architecture.</li>
</ul>

CC BY-SA 4.0
2024-02-24T18:20:44.293Z
Cool!  Does Leela still use MCTS, in conjunction with the new transformer architecture?
Ben G
CC BY-SA 4.0
2024-02-24T18:24:50.633Z
Yeah they do. They've added some improvements to the core UCT formula and to the selfplay data generation process, but the big picture is still very much the same.
KarelPeeters
CC BY-SA 4.0
2024-02-24T12:51:08.990Z
2024-03-02T10:11:03.430Z
2024-03-02T10:11:03.430Z
KarelPeeters
<p>AlphaZero utilized a residual convolutional neural network to estimate move policy and position value.  If it was rebuilt today, would it be more efficient and powerful if they used a transformer architecture instead?  Assume the same amount of compute power and training time is available, and purely self-play, aiming for as fair a comparison as possible.</p>

<reinforcement-learning><convolutional-neural-networks><transformer><alphazero><alphago>
Would AlphaZero perform better if made with transformers?
Ben G
# Would AlphaZero perform better if made with transformers?

(Asked by: username_0 on 2024-02-24)

AlphaZero utilized a residual convolutional neural network to estimate move policy and position value. If it was rebuilt today, would it be more efficient and powerful if they used a transformer architecture instead? Assume the same amount of compute power and training time is available, and purely self-play, aiming for as fair a comparison as possible.

**Tags:** reinforcement-learning, convolutional-neural-networks, transformer, alphazero, alphago


---

## 1 answers

### Answer by username_1 on 2024-02-24. Score: 4

Yes!

[LeelaChessZero](https://github.com/LeelaChessZero/lc0), an open-source re-implementation and continuation of AlphaZero, has been experimenting with this for a while now. Their strongest networks are currently transformers, not convolution-based ResNets anymore.

Some resources:

* The [2022 progress](https://lczero.org/blog/2023/01/2022-progress/#network-architecture) blog post
* More details on the network architecture as of 7 months ago [here](https://github.com/Ergodice/lczero-training/blob/attention-net-body/README.md)
* A [recent comparison](https://lczero.org/blog/2024/02/how-well-do-lc0-networks-compare-to-the-greatest-transformer-network-from-deepmind/) to the latest DeepMind paper, which also uses transformers.
* A [recent blog post](https://lczero.org/blog/2024/02/transformer-progress/) about the current best transformer architecture.

> **Comments:**
> **username_0**: Cool!  Does Leela still use MCTS, in conjunction with the new transformer architecture?  
> **username_1**: Yeah they do. They've added some improvements to the core UCT formula and to the selfplay data generation process, but the big picture is still very much the same.  

---

