Answers.Body: <p>Will put this here as a summary of the solution.<br>
  To get to cartpole reward 500 shouldn't take a modern compute too long (few minutes).
  <br>
  If it runs longer than that it can be because of a lot of ways to speed it up (assuming the code works as it should): <br></p>
  <ol>
  <li>The versions of the libraries can matter a lot, downgrading ended up fixing the problem. (as the creator of the question pointed out).</li>
  <li>find better hyper-parameters. (For example: since DQN is quite unstable, one can play around with the syncing hyper-parameter to make it more stable as that may improve convergence (stability and/or speed))</li>
  <li>use gather, masks and other tricks so you don't have to use very slow python loops.</li>
  <li>use the gpu.</li>
  <li>getting better hardware. (unless you run on an actual potato hardware shouldn't be a problem)</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-13T07:32:51.643Z
Answers.Text: Suggest add 6) disable rendering the game during training. Makes a huge difference in my experience. You can still switch it on for rendering a few test episodes
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-18T22:09:29.773Z
Answers.LastActivityDate: 2022-07-18T22:09:29.773Z
Answers.UserDisplayName: hal9000
Body: <p>I am trying to solve the <a href="https://www.gymlibrary.ml/environments/classic_control/cart_pole" rel="nofollow noreferrer">cartpole environment</a> (<a href="https://github.com/openai/gym/blob/master/gym/envs/classic_control/cartpole.py" rel="nofollow noreferrer">GitHub</a>) using DQN agent. I have been building my own DQN agent by following a tutorial by Jon Krohn.</p>
  <p>I am able to solve the environment with a maximum reward of 499. I am using the experience replay method, and I have given the number of episodes as 1000 episodes.</p>
  <p>In the tutorial it takes very much less time to solve the environment, whereas for me it's taking almost 2 hours to solve it.</p>
  <p>The library I am using for building my model is tensorflow (tf.keras).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-16T11:10:52.617Z
Comments.Text: Could you share your code?
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-16T15:19:36.490Z
Comments.Text: Hi @hal9000 , i am sharing the github link here. This is the code I used.  https://github.com/the-deep-learners/TensorFlow-LiveLessons/blob/master/notebooks/cartpole_dqn.ipynb  . I just want to know how much time it takes to solve
Comments.UserDisplayName: Rishidhar Kasam
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-16T16:17:01.713Z
Comments.Text: From my own experience it takes around 5 min to get maximum reward on cartpole. 2 hours seems indeed a bit long.
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-16T16:40:23.630Z
Comments.Text: Looking at your code I think it might be because you extract the mini batch and process each part separately. (there are nice ways to process the entire batch at once, making it go faster). Also I think I see you update the network that is used in the prediction directly instead of resyncing every m steps MSE(Q(state,action,param),reward+gamma*Q'(next_state,action,frozen_param)), while it might not cause instability for cartpole, you might want to delay resyncing the frozen parameters with the current parameters as that makes convergences more stable.
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-16T21:29:54.480Z
Comments.Text: Thanks for your comment @hal9000 , I understand that this code needs improvement but shouldn't it be able to run in less than 10 minutes ? Can you please help me with the dependencies that you used when you tried to solve this environment along with the versions ? And also, I use anaconda, so does that contribute to this problem ? I am pretty sure it doesn't but I would like to know what you think about it. Thanks
Comments.UserDisplayName: Rishidhar Kasam
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-17T07:02:07.680Z
Comments.Text: I also use anaconda but I coded it in pytorch and I also run it on the gpu.
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-17T08:04:42.367Z
Comments.Text: honestly there are a lot of things that can make the code run faster: making the algorithm more stable (dqn tends to overestimate the Q-values), having faster hardware, running things in parallel, running on the gpu, using the right choice of hyper parameters, maybe having a slightly bigger network, using vectorization, reduce the amount of prints (I use tqdm as loading bar for the epochs).
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-17T08:17:27.867Z
Comments.Text: Try to avoid using pure python as it makes things realllyyy slow. Use gather like operations for selecting the Q values from the online network. Use masking instead of if statements like Q_target[dones] = 0.
Comments.UserDisplayName: hal9000
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-18T08:06:51.660Z
Comments.Text: Thanks @hal9000 , I will try to use whatever you have suggested. I don't think my hardware is a problem since my processor is i7-11800h with a nvidia rtx 3070. I did not run it on gpu since it did not have any images and I thought that running it on cpu would be enough. Anyways, thanks for your help.
Comments.UserDisplayName: Rishidhar Kasam
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-18T11:06:26.173Z
Comments.Text: Hi @hal9000 , the problem has been solved. Thanks for your help. Apparently, it was an issue with the versions of my libraries and it looks like the most recent versions were the problem. I downgraded them and as you said, it is done within a couple of minutes. Thanks again.
Comments.UserDisplayName: Rishidhar Kasam
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-18T21:34:19.913Z
Comments.Text: no problem, I'm glad I could help. :)
Comments.UserDisplayName: hal9000
Tags: <reinforcement-learning><tensorflow><dqn><keras><gym>
Title: Time taken to solve cartpole environment using DQN
UserDisplayName: Rishidhar Kasam
ThreadText: # Time taken to solve cartpole environment using DQN
  
  (Asked by: username_0 on 2022-07-15)
  
  I am trying to solve the [cartpole environment](https://www.gymlibrary.ml/environments/classic_control/cart_pole) ([GitHub](https://github.com/openai/gym/blob/master/gym/envs/classic_control/cartpole.py)) using DQN agent. I have been building my own DQN agent by following a tutorial by Jon Krohn.
  
  I am able to solve the environment with a maximum reward of 499. I am using the experience replay method, and I have given the number of episodes as 1000 episodes.
  
  In the tutorial it takes very much less time to solve the environment, whereas for me it's taking almost 2 hours to solve it.
  
  The library I am using for building my model is tensorflow (tf.keras).
  
  **Tags:** reinforcement-learning, tensorflow, dqn, keras, gym
  > **Comments:**
  > **username_1**: Could you share your code?  
  > **username_0**: Hi @hal9000 , i am sharing the github link here. This is the code I used.  https://github.com/the-deep-learners/TensorFlow-LiveLessons/blob/master/notebooks/cartpole_dqn.ipynb  . I just want to know how much time it takes to solve  
  > **username_1**: From my own experience it takes around 5 min to get maximum reward on cartpole. 2 hours seems indeed a bit long.  
  > **username_1**: Looking at your code I think it might be because you extract the mini batch and process each part separately. (there are nice ways to process the entire batch at once, making it go faster). Also I think I see you update the network that is used in the prediction directly instead of resyncing every m steps MSE(Q(state,action,param),reward+gamma*Q'(next_state,action,frozen_param)), while it might not cause instability for cartpole, you might want to delay resyncing the frozen parameters with the current parameters as that makes convergences more stable.  
  > **username_0**: Thanks for your comment @hal9000 , I understand that this code needs improvement but shouldn't it be able to run in less than 10 minutes ? Can you please help me with the dependencies that you used when you tried to solve this environment along with the versions ? And also, I use anaconda, so does that contribute to this problem ? I am pretty sure it doesn't but I would like to know what you think about it. Thanks  
  > **username_1**: I also use anaconda but I coded it in pytorch and I also run it on the gpu.  
  > **username_1**: honestly there are a lot of things that can make the code run faster: making the algorithm more stable (dqn tends to overestimate the Q-values), having faster hardware, running things in parallel, running on the gpu, using the right choice of hyper parameters, maybe having a slightly bigger network, using vectorization, reduce the amount of prints (I use tqdm as loading bar for the epochs).  
  > **username_1**: Try to avoid using pure python as it makes things realllyyy slow. Use gather like operations for selecting the Q values from the online network. Use masking instead of if statements like Q_target[dones] = 0.  
  > **username_0**: Thanks @hal9000 , I will try to use whatever you have suggested. I don't think my hardware is a problem since my processor is i7-11800h with a nvidia rtx 3070. I did not run it on gpu since it did not have any images and I thought that running it on cpu would be enough. Anyways, thanks for your help.  
  > **username_0**: Hi @hal9000 , the problem has been solved. Thanks for your help. Apparently, it was an issue with the versions of my libraries and it looks like the most recent versions were the problem. I downgraded them and as you said, it is done within a couple of minutes. Thanks again.  
  > **username_1**: no problem, I'm glad I could help. :)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-18. Score: 0
  
  Will put this here as a summary of the solution.  
  To get to cartpole reward 500 shouldn't take a modern compute too long (few minutes).
    
  If it runs longer than that it can be because of a lot of ways to speed it up (assuming the code works as it should):
  
  1. The versions of the libraries can matter a lot, downgrading ended up fixing the problem. (as the creator of the question pointed out).
  2. find better hyper-parameters. (For example: since DQN is quite unstable, one can play around with the syncing hyper-parameter to make it more stable as that may improve convergence (stability and/or speed))
  3. use gather, masks and other tricks so you don't have to use very slow python loops.
  4. use the gpu.
  5. getting better hardware. (unless you run on an actual potato hardware shouldn't be a problem)
  
  > **Comments:**
  > **username_2**: Suggest add 6) disable rendering the game during training. Makes a huge difference in my experience. You can still switch it on for rendering a few test episodes  
  
  ---
  
  