Body: <p>I'm trying to find the optimal policy for the mountain car problem using deep Q learning with images as input, however, I cannot find a way to get my Q function to give me good solutions (I followed multiple tutorials for similar problems (Atari games and Flappy bird)).
  I'm working on Python with Keras.</p>
  
  <p>The images are given in the following format :</p>
  
  <p><img src="https://i.sstatic.net/hf3Zr.png" alt="input"></p>
  
  <p>400x400 pixels, where the bar on the bottom right corner represents the speed of the car.</p>
  
  <p>To check where my problem might lie, I thought it would be best if I split the problem by first ensuring that I can find a network which would successfully find the state of the car (position and speed, since it's all we need to find it) by feeding my convolutional network images of random states (uniformly distributed within the state space).</p>
  
  <p>After unsuccessful results, I decided to split it even more by only trying to find the two state variable separately.</p>
  
  <p>This is the best kind of result that I get, when the network doesn't predict all the state to be the same (which seem to happen a lot with relu activation).
  The state space is divided in 50x50 matrix to make my predictions. The predicted speed is on the left, and the absolute error is on the right</p>
  
  <p><img src="https://i.sstatic.net/r2QUB.png" alt="state_error">.</p>
  
  <p>The images fed to the network are pre-processed the follow way : 
   1. Gray scale
   2. Resize (I tried 50x50, 100x100 and 150x150)
   3. Values centered around 0 in [-1;1] (this seemed to help a bit with the relu activation)</p>
  
  <p>The network I used to try to find the speed is first a convolution layer (I tried 32 windows of kernel_size=(4,4) (8,8) and (16,16), strides=(1,1) and (2,2), activation= relu, linear, tanh.</p>
  
  <p>Optional additional convolution layers of kernel size half the previous layer.</p>
  
  <p>And an optional last dense layer of dimension 32 or activation relu, linear or tanh.</p>
  
  <p>The output layer is dimension one with linear activation.</p>
  
  <p>The way I train the network is by feeding the fit function with 32 random samples and let the network train for 25 epochs with batch_size 32 and repeat  ad libitum.</p>
  
  <p>It's becoming extremely frustrating, especially with my GPU not fitting the requirement for GPU computation to check if I get some results faster.</p>
  
  <p>Can anyone tell me if I'm doing something wrong that I'm missing and what can I do to improve my method to eventually manage to get the reinforcement algorithm to work ?
  Like the size of the training sample, batch size and epochs of the fit function, the structure of the network, ...</p>
  
  <p>Edit : I finally found a way for my reinforcement learning to converge to the true Q value : each time I run a new episode and put it in my replay memory, I run many fits of different mini-batches. This is something I thought I did by increasing the number of epochs in the parameter of the fit function, but I guess it doesn't work as I thought it would.
  I'm letting it train a bit and then I will try the same method for the sub-problems mentioned above.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-12T22:59:47.527Z
Comments.Text: Welcome to AI!  (I think there is a rendering error in the second set of graphics--I'm seeing the numbers overlap, resulting in illegibility.)
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-13T01:13:23.117Z
Comments.Text: Can you please provide a figure of the resized image? Are you using a sliding window to find the car?
Comments.UserDisplayName: Charles
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-13T08:57:28.717Z
Comments.Text: The solution in your edit is "re-work the experience replay logic", of which there was no hint that this was the problem before your edit . . . Your question is interesting. However, your edit is revealing that the solution is in the details of the project - hyperparameters and implementation details. It is unlikely anyone here can take even the quite long description here and analyse the fault which is essentially "my learning algorithm is performing badly". At best you can expect well-meaning answers that suggest something to check that you may or may not already be doing.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-13T12:57:20.463Z
Comments.Text: "there was no hint that this was the problem" If I knew it was the problem, I wouldn't have looked for help, would I ?;) Basically the reason why my protocol didn't work(or would take ages to converge (maybe because I use CPU ?)) is because I trained only one mini batch at a time in-between each episodes, which is what every algorithm I've looked at seemed to be doing (and why I didn't think about mentioning it). The change I made for it to work (in a reasonable amount of time) is to train over many random mini batches in-between episodes, since creating the image is time consuming.
Comments.UserDisplayName: user3548298
Tags: <deep-learning><python><keras>
Title: Mountain car problem with images - not converging
UserDisplayName: user3548298
ThreadText: # Mountain car problem with images - not converging
  
  (Asked by: username_0 on 2018-05-12)
  
  I'm trying to find the optimal policy for the mountain car problem using deep Q learning with images as input, however, I cannot find a way to get my Q function to give me good solutions (I followed multiple tutorials for similar problems (Atari games and Flappy bird)).
  I'm working on Python with Keras.
  
  The images are given in the following format :
  
  ![input](https://i.sstatic.net/hf3Zr.png)
  
  400x400 pixels, where the bar on the bottom right corner represents the speed of the car.
  
  To check where my problem might lie, I thought it would be best if I split the problem by first ensuring that I can find a network which would successfully find the state of the car (position and speed, since it's all we need to find it) by feeding my convolutional network images of random states (uniformly distributed within the state space).
  
  After unsuccessful results, I decided to split it even more by only trying to find the two state variable separately.
  
  This is the best kind of result that I get, when the network doesn't predict all the state to be the same (which seem to happen a lot with relu activation).
  The state space is divided in 50x50 matrix to make my predictions. The predicted speed is on the left, and the absolute error is on the right
  
  ![state_error](https://i.sstatic.net/r2QUB.png).
  
  The images fed to the network are pre-processed the follow way :
  1. Gray scale
  2. Resize (I tried 50x50, 100x100 and 150x150)
  3. Values centered around 0 in [-1;1] (this seemed to help a bit with the relu activation)
  
  The network I used to try to find the speed is first a convolution layer (I tried 32 windows of kernel\_size=(4,4) (8,8) and (16,16), strides=(1,1) and (2,2), activation= relu, linear, tanh.
  
  Optional additional convolution layers of kernel size half the previous layer.
  
  And an optional last dense layer of dimension 32 or activation relu, linear or tanh.
  
  The output layer is dimension one with linear activation.
  
  The way I train the network is by feeding the fit function with 32 random samples and let the network train for 25 epochs with batch\_size 32 and repeat ad libitum.
  
  It's becoming extremely frustrating, especially with my GPU not fitting the requirement for GPU computation to check if I get some results faster.
  
  Can anyone tell me if I'm doing something wrong that I'm missing and what can I do to improve my method to eventually manage to get the reinforcement algorithm to work ?
  Like the size of the training sample, batch size and epochs of the fit function, the structure of the network, ...
  
  Edit : I finally found a way for my reinforcement learning to converge to the true Q value : each time I run a new episode and put it in my replay memory, I run many fits of different mini-batches. This is something I thought I did by increasing the number of epochs in the parameter of the fit function, but I guess it doesn't work as I thought it would.
  I'm letting it train a bit and then I will try the same method for the sub-problems mentioned above.
  
  **Tags:** deep-learning, python, keras
  
  > **Comments:**
  > **username_1**: Welcome to AI!  (I think there is a rendering error in the second set of graphics--I'm seeing the numbers overlap, resulting in illegibility.)  
  > **username_2**: Can you please provide a figure of the resized image? Are you using a sliding window to find the car?  
  > **username_3**: The solution in your edit is "re-work the experience replay logic", of which there was no hint that this was the problem before your edit . . . Your question is interesting. However, your edit is revealing that the solution is in the details of the project - hyperparameters and implementation details. It is unlikely anyone here can take even the quite long description here and analyse the fault which is essentially "my learning algorithm is performing badly". At best you can expect well-meaning answers that suggest something to check that you may or may not already be doing.  
  > **username_0**: "there was no hint that this was the problem" If I knew it was the problem, I wouldn't have looked for help, would I ?;) Basically the reason why my protocol didn't work(or would take ages to converge (maybe because I use CPU ?)) is because I trained only one mini batch at a time in-between each episodes, which is what every algorithm I've looked at seemed to be doing (and why I didn't think about mentioning it). The change I made for it to work (in a reasonable amount of time) is to train over many random mini batches in-between episodes, since creating the image is time consuming.  
  
  ---
  
  ## 0 answers
  
  