Answers.Body: <p>You should use an algorithm to try doing a solve for the maze optimally, maybe A* algorithm. If the optimal steps is also in the range of your network, your network may have reached it's best. If the optimal step is much less, you can try increasing the step penalty and increasing the reward for reaching the end. Hope you can succeed in this problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-03T11:39:52.087Z
Answers.Text: Hello, thank you for the answer. I am thinking to use a DQN because 1) I am expecting to learn from, say, 1000 images, and has a model which can help to quickly solve the maze on other 5000 images. 2) my input are images of raw pixels. Therefore, I do need certain convolutional layers anyway, right?
Answers.UserDisplayName: lllllllllllll
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-03T13:03:16.620Z
Answers.Text: if you use images, you should use CNN.
Answers.UserDisplayName: Clement
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-03T13:22:03.387Z
Answers.Text: Yes, that's right. The DQN is stacked with three convolutional layers followed by two fully connected layer.
Answers.UserDisplayName: lllllllllllll
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-04T06:40:22.043Z
Answers.Text: If the pathways are 1 pixels apart, the network is optimized already. But if it is wider apart, you may have to change the rewards and penalties.
Answers.UserDisplayName: Clement
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-03T08:00:37.083Z
Answers.LastActivityDate: 2019-06-03T08:00:37.083Z
Answers.UserDisplayName: Clement
Body: <p>I am using deep reinforcement learning to solve a classic maze escaping task, similar to the implementation provided <a href="https://github.com/prasoon2211/pacman-RL/blob/master/pacman_vanilla_DQN/pacman_agent.py" rel="nofollow noreferrer">here</a>, except the following three key differences:</p>
  
  <ol>
  <li><p>instead of using a <code>numpy</code> array as the input of a standard maze escaping task, I am feeding the model with an image at each step; the image is a 1300 * 900 RGB image, so it is not too small.</p></li>
  <li><p>reward: </p>
  
  <ul>
  <li>each valid move has a small negative reward (penalize long move) </li>
  <li>each invalid move has a big negative reward (run into other objects or boundaries) </li>
  <li>Each blocked move has the minimal reward (not common) </li>
  <li>Find the remote detectors’ defect has a positive reward (5)</li>
  </ul></li>
  <li><p>I tweaked the parameters of replay memory, reduced the size of the replay memory buffer.</p></li>
  </ol>
  
  <p>Regarding the implementation, I basically do not change the agent setup except the above items, and I implemented my <code>env</code> to wrap my customized maze. </p>
  
  <p>But the problem is that, the accumulated reward (first 200 rounds of successful escaping) does not increase:</p>
  
  <p><a href="https://i.sstatic.net/kUoJU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kUoJU.png" alt="enter image description here"></a></p>
  
  <p>And the number of steps it takes to escape one maze is also stable somewhat:</p>
  
  <p><a href="https://i.sstatic.net/ZxftI.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZxftI.png" alt="enter image description here"></a></p>
  
  <p>Here are my question, on which aspect I could start to look at to optimize my problem? Or is it still too early and I will need to train more time?</p>
  
Tags: <deep-learning><reinforcement-learning><deep-rl>
Title: Reward does not increase for a maze escaping problem with DQN
UserDisplayName: lllllllllllll
ThreadText: # Reward does not increase for a maze escaping problem with DQN
  
  (Asked by: username_0 on 2019-06-01)
  
  I am using deep reinforcement learning to solve a classic maze escaping task, similar to the implementation provided [here](https://github.com/prasoon2211/pacman-RL/blob/master/pacman_vanilla_DQN/pacman_agent.py), except the following three key differences:
  
  1. instead of using a `numpy` array as the input of a standard maze escaping task, I am feeding the model with an image at each step; the image is a 1300 \* 900 RGB image, so it is not too small.
  2. reward:
  
     * each valid move has a small negative reward (penalize long move)
     * each invalid move has a big negative reward (run into other objects or boundaries)
     * Each blocked move has the minimal reward (not common)
     * Find the remote detectors’ defect has a positive reward (5)
  3. I tweaked the parameters of replay memory, reduced the size of the replay memory buffer.
  
  Regarding the implementation, I basically do not change the agent setup except the above items, and I implemented my `env` to wrap my customized maze.
  
  But the problem is that, the accumulated reward (first 200 rounds of successful escaping) does not increase:
  
  [![enter image description here](https://i.sstatic.net/kUoJU.png)](https://i.sstatic.net/kUoJU.png)
  
  And the number of steps it takes to escape one maze is also stable somewhat:
  
  [![enter image description here](https://i.sstatic.net/ZxftI.png)](https://i.sstatic.net/ZxftI.png)
  
  Here are my question, on which aspect I could start to look at to optimize my problem? Or is it still too early and I will need to train more time?
  
  **Tags:** deep-learning, reinforcement-learning, deep-rl
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-03. Score: 1 | Accepted answer
  
  You should use an algorithm to try doing a solve for the maze optimally, maybe A\* algorithm. If the optimal steps is also in the range of your network, your network may have reached it's best. If the optimal step is much less, you can try increasing the step penalty and increasing the reward for reaching the end. Hope you can succeed in this problem.
  
  > **Comments:**
  > **username_0**: Hello, thank you for the answer. I am thinking to use a DQN because 1) I am expecting to learn from, say, 1000 images, and has a model which can help to quickly solve the maze on other 5000 images. 2) my input are images of raw pixels. Therefore, I do need certain convolutional layers anyway, right?  
  > **username_1**: if you use images, you should use CNN.  
  > **username_0**: Yes, that's right. The DQN is stacked with three convolutional layers followed by two fully connected layer.  
  > **username_1**: If the pathways are 1 pixels apart, the network is optimized already. But if it is wider apart, you may have to change the rewards and penalties.  
  
  ---
  
  