Answers.Body: <blockquote>
    <p>Which do you think is the best method? </p>
  </blockquote>
  
  <p>As with most machine learning, each approach has its strengths and weaknesses, and other than a little bit of intuition:</p>
  
  <ul>
  <li><p>Policy-based methods are strong in large or continuous action spaces, and/or where there is a simple relationship between state and optimal action. E.g. controlling a robotic arm with continuous action space.</p></li>
  <li><p>Value-based methods are strong where there is a simple relationship between state and value under an optimal policy. E.g. in a maze game.</p></li>
  </ul>
  
  <p>It may not always be clear which is best, in which case experimentation is required. If using neural networks, there will then be a large number of hyper-parameters on each approach, so it may be hard to come to a strict conclusion about which is better. Although you can include "easy to find a working model" or "robust for a range of hyper-parameter values" as benefits of any type of model if you wish - these are important practical benefits of any approach, developers rarely want to do 100s of experiments to tune a learning rate parameter for example.</p>
  
  <blockquote>
    <p>Which is the best way to do AI? Or some combination of the two?</p>
  </blockquote>
  
  <p>Actor-Critic, as seen in Asynchronous Advantage Actor Critic. A3C and A2C (deterministic variant of A3C) is producing current state-of-the-art results in video games. This is a combination of both approaches, where the agent maintains two related models - one, the Actor, generates a policy directly by looking at the state, and the second, the Critic, tracks the estimated value of each state. Often, these two models share some parameters - e.g. using neural networks, the initial layers may be the same for both.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-04-16T07:17:01.807Z
Answers.LastActivityDate: 2018-04-16T07:17:01.807Z
Answers.UserDisplayName: Neil Slater
Body: <p>If you have a game and you are training an AI there seems to be two ways to do it.</p>
  
  <p>First you take the game-state and a possible move and evaluate whether this move would be good or bad:</p>
  
  <p>(1) GAME_STATE + POSSIBLE_MOVE  --> Good or bad?</p>
  
  <p>The second is to take the game state and get probabilities of every conceivable move:</p>
  
  <p>(2) GAME_STATE ---> Probabilities for each move</p>
  
  <p>It seems that both models are used. I.e. in language and RNN might use (2) to find the probabilities for each next word or letter. But AlphaZero might use (1). Noting also that in a game like chess GAME_STATE + POSSIBLE_MOVE = NEW_GAME_STATE. Whereas in some games you might not know the result of your move.</p>
  
  <p>Which do you think is the best method? Which is the best way to do AI? Or some combination of the two?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-15T19:59:28.337Z
Comments.Text: Actually AlphaGo uses 1 + 2 (plus a look-ahead search to help refine them), and this is relatively common too (search "Actor-Critic").
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-15T21:57:29.683Z
Comments.Text: Ah, I see! I don't think AlphaZero uses Actor-Critic though?
Comments.UserDisplayName: zooby
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-04-16T07:03:10.183Z
Comments.Text: Yes you are right, it is not Actor-Critic (as far as I understand it). It does however both generate policies directly and evaluate board positions. The tree search (MCTS) is what links the two.
Comments.UserDisplayName: Neil Slater
Tags: <machine-learning><learning-algorithms>
Title: Which is best: evaluation of states or probability of moves?
UserDisplayName: zooby
ThreadText: # Which is best: evaluation of states or probability of moves?
  
  (Asked by: username_0 on 2018-04-15)
  
  If you have a game and you are training an AI there seems to be two ways to do it.
  
  First you take the game-state and a possible move and evaluate whether this move would be good or bad:
  
  (1) GAME\_STATE + POSSIBLE\_MOVE --> Good or bad?
  
  The second is to take the game state and get probabilities of every conceivable move:
  
  (2) GAME\_STATE ---> Probabilities for each move
  
  It seems that both models are used. I.e. in language and RNN might use (2) to find the probabilities for each next word or letter. But AlphaZero might use (1). Noting also that in a game like chess GAME\_STATE + POSSIBLE\_MOVE = NEW\_GAME\_STATE. Whereas in some games you might not know the result of your move.
  
  Which do you think is the best method? Which is the best way to do AI? Or some combination of the two?
  
  **Tags:** machine-learning, learning-algorithms
  > **Comments:**
  > **username_1**: Actually AlphaGo uses 1 + 2 (plus a look-ahead search to help refine them), and this is relatively common too (search "Actor-Critic").  
  > **username_0**: Ah, I see! I don't think AlphaZero uses Actor-Critic though?  
  > **username_1**: Yes you are right, it is not Actor-Critic (as far as I understand it). It does however both generate policies directly and evaluate board positions. The tree search (MCTS) is what links the two.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-04-16. Score: 1 | Accepted answer
  
  > Which do you think is the best method?
  
  As with most machine learning, each approach has its strengths and weaknesses, and other than a little bit of intuition:
  
  * Policy-based methods are strong in large or continuous action spaces, and/or where there is a simple relationship between state and optimal action. E.g. controlling a robotic arm with continuous action space.
  * Value-based methods are strong where there is a simple relationship between state and value under an optimal policy. E.g. in a maze game.
  
  It may not always be clear which is best, in which case experimentation is required. If using neural networks, there will then be a large number of hyper-parameters on each approach, so it may be hard to come to a strict conclusion about which is better. Although you can include "easy to find a working model" or "robust for a range of hyper-parameter values" as benefits of any type of model if you wish - these are important practical benefits of any approach, developers rarely want to do 100s of experiments to tune a learning rate parameter for example.
  
  > Which is the best way to do AI? Or some combination of the two?
  
  Actor-Critic, as seen in Asynchronous Advantage Actor Critic. A3C and A2C (deterministic variant of A3C) is producing current state-of-the-art results in video games. This is a combination of both approaches, where the agent maintains two related models - one, the Actor, generates a policy directly by looking at the state, and the second, the Critic, tracks the estimated value of each state. Often, these two models share some parameters - e.g. using neural networks, the initial layers may be the same for both.
  
  
  ---
  
  