Body: <p>In most of RL algorithms I saw, there is a <strong>coefficient that reduces</strong> actions exploration over time, to help convergence.</p>
  
  <p>But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous action spaces, the different implementation I saw (mainly <strong>using Ornstein Uhlenbeck process</strong>) is correlated over time, but not decreased.</p>
  
  <p>The action noises are clipped into a range of [-1, 1] and are added to policies that are between [-1, 1] too. So, I don't understand how it could work in environments with hard-to-obtain rewards.</p>
  
  <p>Any thought about this ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-30T16:34:03.837Z
Comments.Text: what is stopping you from multiplying noise value with some decaying coefficient ?
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-30T16:42:38.597Z
Comments.Text: I tried on some openAI environments (Pendulum and BipedalWalker) but could not get any convergence, where non-decreased OU noise made perfectly the job. I can barely understand this, so I should try a lot of different decay coefficient to be sure. May take a long time, so I was expecting this question had an already known answer
Comments.UserDisplayName: Loheek
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-30T16:49:57.123Z
Comments.Text: I'm not sure I don't think it's set in stone it probably depends on environment. You could try experimenting with different setups. Maybe you were decaying exploration too fast, try having it non-decayed for some fixed number of steps and then decayed slowly from that point, or maybe when you see your performance is getting better  and you are getting a lot of good episode outcomes try decreasing it from that point
Comments.UserDisplayName: Brale
Tags: <deep-learning><reinforcement-learning><actor-critic-methods><ddpg>
Title: Should noise (such as OU) be decreased over time in actor / critic algorithms?
UserDisplayName: Loheek
ThreadText: # Should noise (such as OU) be decreased over time in actor / critic algorithms?
  
  (Asked by: username_0 on 2019-04-30)
  
  In most of RL algorithms I saw, there is a **coefficient that reduces** actions exploration over time, to help convergence.
  
  But in Actor-Critic, or other algorithms (A3C, DDPG, ...) used in continuous action spaces, the different implementation I saw (mainly **using Ornstein Uhlenbeck process**) is correlated over time, but not decreased.
  
  The action noises are clipped into a range of [-1, 1] and are added to policies that are between [-1, 1] too. So, I don't understand how it could work in environments with hard-to-obtain rewards.
  
  Any thought about this ?
  
  **Tags:** deep-learning, reinforcement-learning, actor-critic-methods, ddpg
  
  > **Comments:**
  > **username_1**: what is stopping you from multiplying noise value with some decaying coefficient ?  
  > **username_0**: I tried on some openAI environments (Pendulum and BipedalWalker) but could not get any convergence, where non-decreased OU noise made perfectly the job. I can barely understand this, so I should try a lot of different decay coefficient to be sure. May take a long time, so I was expecting this question had an already known answer  
  > **username_1**: I'm not sure I don't think it's set in stone it probably depends on environment. You could try experimenting with different setups. Maybe you were decaying exploration too fast, try having it non-decayed for some fixed number of steps and then decayed slowly from that point, or maybe when you see your performance is getting better  and you are getting a lot of good episode outcomes try decreasing it from that point  
  
  ---
  
  ## 0 answers
  
  