Body: <p>I built a simple AI system that tries to solve the 8 puzzle using DQN.
  The problem is, if the agent gets only a reward greater than zero when winning, the training will take a long time, so I made a smooth reward function instead: <span class="math-container">$R=(n/9)^3$</span>, where <span class="math-container">$n$</span> is the number of pieces that are in the right position.</p>
  
  <p>The training became quicker but the AI chose to match 7 pieces out of 9 to get a reward of <span class="math-container">$(7/9)^3/(1-\gamma) = 0.47/(1-\gamma) = 4.7$</span>, for <span class="math-container">$\gamma=0.9$</span>, choosing to win and getting reward of 1 doesn't make sense to the AI, lowering <span class="math-container">$\gamma$</span> will result in the AI to choose instant reward instead of long-term reward, so that will not be very helpful; lowering rewards of non-winning stats will make the training very slow.</p>
  
  <p>So, how do I choose a good reward function?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-09T19:03:23.817Z
Comments.Text: Try using Manhattan distance between current position and supposed position for all pieces, use that as negative reward.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-10T03:05:44.957Z
Comments.Text: Increase the reward for winning. The reward for winning should normally dwarf all other rewards.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-10T16:59:02.183Z
Comments.Text: Changed the reward of winning to 5, that didn't improve the results a lot somehow. i like the idea of negative rewards i will give it a go.
Comments.UserDisplayName: j4at
Tags: <machine-learning><dqn>
Title: DQN, how to choose the reward fucntion?
UserDisplayName: j4at
ThreadText: # DQN, how to choose the reward fucntion?
  
  (Asked by: username_0 on 2019-12-09)
  
  I built a simple AI system that tries to solve the 8 puzzle using DQN.
  The problem is, if the agent gets only a reward greater than zero when winning, the training will take a long time, so I made a smooth reward function instead: $R=(n/9)^3$, where $n$ is the number of pieces that are in the right position.
  
  The training became quicker but the AI chose to match 7 pieces out of 9 to get a reward of $(7/9)^3/(1-\gamma) = 0.47/(1-\gamma) = 4.7$, for $\gamma=0.9$, choosing to win and getting reward of 1 doesn't make sense to the AI, lowering $\gamma$ will result in the AI to choose instant reward instead of long-term reward, so that will not be very helpful; lowering rewards of non-winning stats will make the training very slow.
  
  So, how do I choose a good reward function?
  
  **Tags:** machine-learning, dqn
  
  > **Comments:**
  > **username_1**: Try using Manhattan distance between current position and supposed position for all pieces, use that as negative reward.  
  > **username_2**: Increase the reward for winning. The reward for winning should normally dwarf all other rewards.  
  > **username_0**: Changed the reward of winning to 5, that didn't improve the results a lot somehow. i like the idea of negative rewards i will give it a go.  
  
  ---
  
  ## 0 answers
  
  