Answers.Body: <p>Interesting project. First thing I'd do is normalize your state by the maximum cell value. This way you can represent multiple situations at once (eg A grid of all 4s and 8s would look the same as a grid of all 16s and 32s).
  Also making the reward = max_cell/2048 might do better as ActorCritic methods seem to do better with rewards within 0-1. </p>
  
  <p>Another reward setup is giving +1 per timestep. It's simple but it also means that maximizing the time to stay alive is the best, which is what I end up doing most of the time when I play anyway. </p>
  
  <p>Good luck!</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-05-09T15:27:49.717Z
Answers.LastActivityDate: 2017-05-09T15:27:49.717Z
Answers.UserDisplayName: Jaden Travnik
Body: <p>I implemented the Actor-Critic with n-step TD prediction to learn to play <a href="http://2048game.com/" rel="nofollow noreferrer">the 2048 game</a></p>
  <p>For the environment, I don't use <a href="https://2048game.com/" rel="nofollow noreferrer">this 2048 implementation</a>. I use a simple one without any graphical interface, just pure matrices. The input for the neural network is the log2 of the game board.</p>
  <p>The structure of my neural network is:</p>
  <ol>
  <li>Input layer</li>
  <li>Hidden layer with 16 units</li>
  <li>Softmax layer with 4 units (up, down, left, right) for the actor</li>
  <li>Linear regression for the critic</li>
  </ol>
  <p>The hidden layer is shared between the actor and critic, so the same hidden layer (point 2) is connected to both the softmax layer of the actor and the linear regression layer of the critic.</p>
  <p>The reward in the original game is the value of the merged cells. For example, if two 4s merged, then the reward is 8. My reward function is almost the same, except I take the log2 of it.</p>
  <p>I tried these parameters and I also tweaked the learning rate, the <span class="math-container">$\gamma$</span>, but I couldn't achieve any good result.</p>
  <p>Could you recommend what should I change?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-12T13:18:03.493Z
Comments.Text: This is an old question, but maybe you should have specified which values for the learning rate and gamma that you used and maybe you should have provided a plot that describes the results/performance of the RL agent.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><game-ai><deep-rl><actor-critic-methods><hyper-parameters>
Title: Why can't my implementation of the Actor-Critic algorithm achieve good results in the 2048 game?
UserDisplayName: Molnár István
ThreadText: # Why can't my implementation of the Actor-Critic algorithm achieve good results in the 2048 game?
  
  (Asked by: username_0 on 2017-05-09)
  
  I implemented the Actor-Critic with n-step TD prediction to learn to play [the 2048 game](http://2048game.com/)
  
  For the environment, I don't use [this 2048 implementation](https://2048game.com/). I use a simple one without any graphical interface, just pure matrices. The input for the neural network is the log2 of the game board.
  
  The structure of my neural network is:
  
  1. Input layer
  2. Hidden layer with 16 units
  3. Softmax layer with 4 units (up, down, left, right) for the actor
  4. Linear regression for the critic
  
  The hidden layer is shared between the actor and critic, so the same hidden layer (point 2) is connected to both the softmax layer of the actor and the linear regression layer of the critic.
  
  The reward in the original game is the value of the merged cells. For example, if two 4s merged, then the reward is 8. My reward function is almost the same, except I take the log2 of it.
  
  I tried these parameters and I also tweaked the learning rate, the $\gamma$, but I couldn't achieve any good result.
  
  Could you recommend what should I change?
  
  **Tags:** reinforcement-learning, game-ai, deep-rl, actor-critic-methods, hyper-parameters
  
  > **Comments:**
  > **username_1**: This is an old question, but maybe you should have specified which values for the learning rate and gamma that you used and maybe you should have provided a plot that describes the results/performance of the RL agent.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-05-09. Score: 1
  
  Interesting project. First thing I'd do is normalize your state by the maximum cell value. This way you can represent multiple situations at once (eg A grid of all 4s and 8s would look the same as a grid of all 16s and 32s).
  Also making the reward = max\_cell/2048 might do better as ActorCritic methods seem to do better with rewards within 0-1.
  
  Another reward setup is giving +1 per timestep. It's simple but it also means that maximizing the time to stay alive is the best, which is what I end up doing most of the time when I play anyway.
  
  Good luck!
  
  
  ---
  
  