Answers.Body: <p>You can actually combine AlphaZero-like approach with DQN:
  <a href="https://github.com/imagry/aleph_star" rel="nofollow noreferrer">A* + DQN</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-28T17:10:26.657Z
Answers.Text: The A* algorithm in that link is the pathfinding / graph search algorithm, that's something completely different from AlphaZero (or even the AlphaStar program for StarCraft)
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-01T07:40:11.843Z
Answers.Text: It has a lot in  common with AlphaZero. Both use tree search for generating training  samples and deep network for action evaluation at the node. The main difference is  AlephZero use Monte Carlo tree search and this algo use A* tree search. The common main idea is that tree branching allow reduce extrapolation error of RL methods. AlphaStar on the other hand has little in common with AlphaZero. It's just policy gradient algorithm
Answers.UserDisplayName: mirror2image
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-28T13:04:20.260Z
Answers.LastActivityDate: 2019-02-28T13:04:20.260Z
Answers.UserDisplayName: mirror2image
Answers.Body: <p>DQN and AlphaZero do not share much in terms of implementation.</p>
  <p>However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, value, policy, then these are interchangeable between DQN and AlphaZero. When it comes to implementation, and what each part of the system is doing, then this is less interchangeable. For instance two networks you have read about in AlphaZero are the policy network and value network. Whilst double DQN alternates between two value networks.</p>
  <p>Probably the best resource that summarises both DQN and AlphaZero, and explains how they extend the basic RL framework in different ways is Sutton &amp; Barto's <a href="http://www.incompleteideas.net/book/RLbook2018.pdf" rel="noreferrer">Reinforcement Learning: An Introduction (second edition)</a> - Chapter 16 sections 5 and 6 cover the designs of DQN Atari, AlphaGo and AlphaZero in some depth.</p>
  <p>In brief:</p>
  <h2>DQN Atari</h2>
  <ul>
  <li>Is model-free</li>
  <li>Uses an action value estimator for <span class="math-container">$Q(s,a)$</span> values, based on a Convolutional Neural Network (CNN)</li>
  <li>Uses experience replay and temporarily frozen target network to stabilise learning process</li>
  <li>Uses a variety of tricks to simplify and standardise the state description and reward structure so that the exact same design and hyperparameters work across multiple games, demonstrating that it is a general learner.</li>
  </ul>
  <h2>AlphaZero</h2>
  <ul>
  <li>Is model based (although <em>some</em> of the learning is technically model-free, based on samples of play)</li>
  <li>Uses a policy network (estimating <span class="math-container">$\pi(a|s)$</span>) and a state value network (estimating <span class="math-container">$V(s)$</span>), based on CNNs. In practice for efficiency the NN for these share many layers and parameters, so how many &quot;networks&quot; there are depends how you want to count them.
  <ul>
  <li>The earlier AlphaGo version had 4 separate networks, 3 variations of policy network - used during play at different stages of planning - and one value network.</li>
  </ul>
  </li>
  <li>Is designed around self-play</li>
  <li>Uses Monte Carlo Tree Search (MCTS) as part of estimating returns - MCTS is a planning algorithm critical to AlphaZero's success, and there is no equivalent component in DQN</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-13T09:16:49.687Z
Answers.Text: @DennisSoemers: I just re-read the chapters in Sutton & Barto. From what I understand of it, there were 4 separate networks in AlphaGo (3 variations of Policy and one Value network), and just one combined network in AlphaGo Zero.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-13T09:56:12.503Z
Answers.Text: Ah yeah that's right, AlphaGo had the SL policy network, RL policy network, and Value network. Well, and the rollout policy, but I don't count that as a network (it's just linear + softmax). There was something about the SL policy network vs RL policy network mentioned in the paper, can't remember 100% for sure what it was. Think it was something like that the RL Policy network was better for generating experience to train the Value Network, but the SL policy network was still better at biasing the Selection phase (due to having higher entropy). Something like that... but not 100% sure
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-28T17:11:34.097Z
Answers.Text: Note that in practice we usually just use a single network for the policy $\pi$ as well as the value estimates $V$, just with different heads for the different outputs (most of the weights are shared)
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-28T21:08:01.277Z
Answers.Text: @DennisSoemers: Yes. IIRC, there is a second simplified policy network as well in AlphaGo Zero, in order to run a "rollout policy" quickly. I'm skating over the detail a little.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-01T09:08:00.537Z
Answers.Text: I think that was only AlphaGo, not AlphaGo Zero anymore :) I'm quite sure the simplified policy in AlphaGo was not even really a network anymore, just a linear function + softmax. AlphaGo Zero doesn't really do rollouts anymore, so it doesn't need that either.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-27T08:08:12.353Z
Answers.LastActivityDate: 2019-03-13T09:19:57.707Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have already implemented a relatively simple DQN on Pacman.</p>
  
  <p>Now I would like to clearly understand the difference between a DQN and the techniques used by AlphaGo zero/AlphaZero and I couldn't find a place where the features of both approaches are compared. </p>
  
  <p>Also sometimes, when reading through blogs, I believe different terms might in fact be the same mathematical tool which adds to the difficulty of clearly understanding the differences. For example, variations of DQN e.g. Double DQN also uses two networks like alpha zero.</p>
  
  <p>Has someone a good reference regarding this question ? Be it a book or an online ressource.</p>
  
Tags: <reinforcement-learning><dqn><alphazero><deep-rl><alphago-zero>
Title: What is the difference between DQN and AlphaGo Zero?
UserDisplayName: FenryrMKIII
ThreadText: # What is the difference between DQN and AlphaGo Zero?
  
  (Asked by: username_0 on 2019-02-27)
  
  I have already implemented a relatively simple DQN on Pacman.
  
  Now I would like to clearly understand the difference between a DQN and the techniques used by AlphaGo zero/AlphaZero and I couldn't find a place where the features of both approaches are compared.
  
  Also sometimes, when reading through blogs, I believe different terms might in fact be the same mathematical tool which adds to the difficulty of clearly understanding the differences. For example, variations of DQN e.g. Double DQN also uses two networks like alpha zero.
  
  Has someone a good reference regarding this question ? Be it a book or an online ressource.
  
  **Tags:** reinforcement-learning, dqn, alphazero, deep-rl, alphago-zero
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2019-02-27. Score: 6 | Accepted answer
  
  DQN and AlphaZero do not share much in terms of implementation.
  
  However, they are based on the same Reinforcement Learning (RL) theoretical framework. If you understand terms like MDP, reward, return, value, policy, then these are interchangeable between DQN and AlphaZero. When it comes to implementation, and what each part of the system is doing, then this is less interchangeable. For instance two networks you have read about in AlphaZero are the policy network and value network. Whilst double DQN alternates between two value networks.
  
  Probably the best resource that summarises both DQN and AlphaZero, and explains how they extend the basic RL framework in different ways is Sutton & Barto's [Reinforcement Learning: An Introduction (second edition)](http://www.incompleteideas.net/book/RLbook2018.pdf) - Chapter 16 sections 5 and 6 cover the designs of DQN Atari, AlphaGo and AlphaZero in some depth.
  
  In brief:
  
  ## DQN Atari
  
  * Is model-free
  * Uses an action value estimator for $Q(s,a)$ values, based on a Convolutional Neural Network (CNN)
  * Uses experience replay and temporarily frozen target network to stabilise learning process
  * Uses a variety of tricks to simplify and standardise the state description and reward structure so that the exact same design and hyperparameters work across multiple games, demonstrating that it is a general learner.
  
  ## AlphaZero
  
  * Is model based (although *some* of the learning is technically model-free, based on samples of play)
  * Uses a policy network (estimating $\pi(a|s)$) and a state value network (estimating $V(s)$), based on CNNs. In practice for efficiency the NN for these share many layers and parameters, so how many "networks" there are depends how you want to count them.
    + The earlier AlphaGo version had 4 separate networks, 3 variations of policy network - used during play at different stages of planning - and one value network.
  * Is designed around self-play
  * Uses Monte Carlo Tree Search (MCTS) as part of estimating returns - MCTS is a planning algorithm critical to AlphaZero's success, and there is no equivalent component in DQN
  
  > **Comments:**
  > **username_2**: Note that in practice we usually just use a single network for the policy $\pi$ as well as the value estimates $V$, just with different heads for the different outputs (most of the weights are shared)  
  > **username_3**: @DennisSoemers: Yes. IIRC, there is a second simplified policy network as well in AlphaGo Zero, in order to run a "rollout policy" quickly. I'm skating over the detail a little.  
  > **username_2**: I think that was only AlphaGo, not AlphaGo Zero anymore :) I'm quite sure the simplified policy in AlphaGo was not even really a network anymore, just a linear function + softmax. AlphaGo Zero doesn't really do rollouts anymore, so it doesn't need that either.  
  > **username_3**: @DennisSoemers: I just re-read the chapters in Sutton & Barto. From what I understand of it, there were 4 separate networks in AlphaGo (3 variations of Policy and one Value network), and just one combined network in AlphaGo Zero.  
  > **username_2**: Ah yeah that's right, AlphaGo had the SL policy network, RL policy network, and Value network. Well, and the rollout policy, but I don't count that as a network (it's just linear + softmax). There was something about the SL policy network vs RL policy network mentioned in the paper, can't remember 100% for sure what it was. Think it was something like that the RL Policy network was better for generating experience to train the Value Network, but the SL policy network was still better at biasing the Selection phase (due to having higher entropy). Something like that... but not 100% sure  
  
  ---
  
  ### Answer by username_1 on 2019-02-28. Score: 1
  
  You can actually combine AlphaZero-like approach with DQN:
  [A\* + DQN](https://github.com/imagry/aleph_star)
  
  > **Comments:**
  > **username_2**: The A* algorithm in that link is the pathfinding / graph search algorithm, that's something completely different from AlphaZero (or even the AlphaStar program for StarCraft)  
  > **username_1**: It has a lot in  common with AlphaZero. Both use tree search for generating training  samples and deep network for action evaluation at the node. The main difference is  AlephZero use Monte Carlo tree search and this algo use A* tree search. The common main idea is that tree branching allow reduce extrapolation error of RL methods. AlphaStar on the other hand has little in common with AlphaZero. It's just policy gradient algorithm  
  
  ---
  
  