Body: <p>I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-29T11:59:39.343Z
Comments.Text: Hi, I hope you are doing great. I tried to update the critic through the noisy actions and the actor through the action its self. However, the actor is not learning, the only thing that changed is that the actions are shifted like all actions become shifted by 2N then 5N and so on. Please not that I am applying the noise for some episodes.
Comments.UserDisplayName: Ahmad Fares
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-29T14:44:10.643Z
Comments.Text: There are many things that can go wrong, I would advise you to study pseudocode  and read [DDPG](https://arxiv.org/pdf/1509.02971.pdf) paper. After that you can study this [github repo](https://github.com/openai/spinningup), it has beginner friendly implementations of some RL algorithms, including DDPG, in PyTorch and TensorFlow 1.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-01T17:29:58.557Z
Comments.Text: Appreciate your help and thank you for the continous support
Comments.UserDisplayName: Ahmad Fares
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-23T15:23:34.277Z
Comments.Text: Do you understand how does epsilon-greedy help exploration in Q-learning for example? Because the idea is basically the same.
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-24T07:13:19.930Z
Comments.Text: Yes, in Q-learning you have discrete actions and we choose actions randomly during exploration and then we choose the action that gives the highest value. Let me explain my confusion by this example. Lets say my actor network generated a force of 1 N, but I need 10 N for the same state so I added big noise to make the action almost 10 N. Now the 10 N and the state go into the critic network and it will give a good value and the actor will know that 1 N is enough, but it will not know that I need 10 not 1.
Comments.UserDisplayName: Ahmad Fares
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-24T16:43:24.250Z
Comments.Text: That's not true because you're not updating actor through action that was modified by noise, you're updating it through action that it wanted to initially take. So in your case the update would be through action 1N and not 10N. Only critic is updated through noise modified action. The reason why this helps exploration is because the critic will eventually get more accurate in representing state-action values so it will "criticize" the actor better through updates
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-25T13:57:22.033Z
Comments.Text: Got it ! I will try it out and revert back to you. Thank you
Comments.UserDisplayName: Ahmad Fares
Tags: <deep-learning><reinforcement-learning><deep-rl><ddpg>
Title: How does adding noise to the action in DDPG help in learning?
UserDisplayName: Ahmad Fares
ThreadText: # How does adding noise to the action in DDPG help in learning?
  
  (Asked by: username_0 on 2020-02-23)
  
  I can't understand how playing with the action generated by the actor network in DDPG by adding the noise term helps in exploration.
  
  **Tags:** deep-learning, reinforcement-learning, deep-rl, ddpg
  
  > **Comments:**
  > **username_1**: Do you understand how does epsilon-greedy help exploration in Q-learning for example? Because the idea is basically the same.  
  > **username_0**: Yes, in Q-learning you have discrete actions and we choose actions randomly during exploration and then we choose the action that gives the highest value. Let me explain my confusion by this example. Lets say my actor network generated a force of 1 N, but I need 10 N for the same state so I added big noise to make the action almost 10 N. Now the 10 N and the state go into the critic network and it will give a good value and the actor will know that 1 N is enough, but it will not know that I need 10 not 1.  
  > **username_1**: That's not true because you're not updating actor through action that was modified by noise, you're updating it through action that it wanted to initially take. So in your case the update would be through action 1N and not 10N. Only critic is updated through noise modified action. The reason why this helps exploration is because the critic will eventually get more accurate in representing state-action values so it will "criticize" the actor better through updates  
  > **username_0**: Got it ! I will try it out and revert back to you. Thank you  
  > **username_0**: Hi, I hope you are doing great. I tried to update the critic through the noisy actions and the actor through the action its self. However, the actor is not learning, the only thing that changed is that the actions are shifted like all actions become shifted by 2N then 5N and so on. Please not that I am applying the noise for some episodes.  
  > **username_1**: There are many things that can go wrong, I would advise you to study pseudocode  and read [DDPG](https://arxiv.org/pdf/1509.02971.pdf) paper. After that you can study this [github repo](https://github.com/openai/spinningup), it has beginner friendly implementations of some RL algorithms, including DDPG, in PyTorch and TensorFlow 1.  
  > **username_0**: Appreciate your help and thank you for the continous support  
  
  ---
  
  ## 0 answers
  
  