Body: <p>In section 3 of the paper <a href="https://arxiv.org/pdf/1509.02971.pdf" rel="noreferrer">Continuous control with deep reinforcement learning</a>, the authors write</p>
  <blockquote>
  <p>As detailed in the supplementary materials we used an <strong>Ornstein-Uhlenbeck process</strong> (Uhlenbeck &amp; Ornstein, 1930) to generate temporally correlated exploration for exploration efficiency in physical control problems with inertia (similar use of autocorrelated noise was introduced in (Wawrzynski, 2015)).</p>
  </blockquote>
  <p>In section 7, they write</p>
  <blockquote>
  <p>For the exploration noise process we used temporally correlated noise in order to explore well in physical environments that have momentum. We used an <strong>Ornstein-Uhlenbeck process</strong> (Uhlenbeck &amp; Ornstein, 1930) with θ = 0.15 and σ = 0.2. The Ornstein-Uhlenbeck process models the velocity of a Brownian particle with friction, which results in temporally correlated values centered around 0.</p>
  </blockquote>
  <p>In a few words, what is the Ornstein-Uhlenbeck process? How does it work? How exactly is it used in DDPG?</p>
  <p>I want to implement the Deep Deterministic Policy Gradient algorithm, and, in the initial actions, noise has to be added. However, I cannot understand how this Ornstein-Uhlenbeck process works. I have searched the internet, but I have not understood the information that I found.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-01T09:47:29.077Z
Comments.Text: It should be noted that more recent work suggests that uncorrelated Gaussian noise works just as well.
  TD3 paper (https://arxiv.org/pdf/1802.09477.pdf): "Unlike the original DDPG, we used uncorrelated noise for exploration as we found noise drawn from the Ornstein-Uhlenbeck (Uhlenbeck & Ornstein, 1930) process offered no performance benefits."
  D4PG paper (https://arxiv.org/pdf/1804.08617.pdf): "We experimented with correlated noise drawn from an Ornstein-Uhlenbeck process, as suggested by (Lillicrap et al., 2016), however we found this was unnecessary and did not add to performance."
Comments.UserDisplayName: Jonas De Schouwer
Tags: <reinforcement-learning><deep-rl><policy-gradients><papers><ddpg>
Title: How does the Ornstein-Uhlenbeck process work, and how it is used in DDPG?
UserDisplayName: dani
ThreadText: # How does the Ornstein-Uhlenbeck process work, and how it is used in DDPG?
  
  (Asked by: username_0 on 2020-08-21)
  
  In section 3 of the paper [Continuous control with deep reinforcement learning](https://arxiv.org/pdf/1509.02971.pdf), the authors write
  
  > As detailed in the supplementary materials we used an **Ornstein-Uhlenbeck process** (Uhlenbeck & Ornstein, 1930) to generate temporally correlated exploration for exploration efficiency in physical control problems with inertia (similar use of autocorrelated noise was introduced in (Wawrzynski, 2015)).
  
  In section 7, they write
  
  > For the exploration noise process we used temporally correlated noise in order to explore well in physical environments that have momentum. We used an **Ornstein-Uhlenbeck process** (Uhlenbeck & Ornstein, 1930) with θ = 0.15 and σ = 0.2. The Ornstein-Uhlenbeck process models the velocity of a Brownian particle with friction, which results in temporally correlated values centered around 0.
  
  In a few words, what is the Ornstein-Uhlenbeck process? How does it work? How exactly is it used in DDPG?
  
  I want to implement the Deep Deterministic Policy Gradient algorithm, and, in the initial actions, noise has to be added. However, I cannot understand how this Ornstein-Uhlenbeck process works. I have searched the internet, but I have not understood the information that I found.
  
  **Tags:** reinforcement-learning, deep-rl, policy-gradients, papers, ddpg
  > **Comments:**
  > **username_1**: It should be noted that more recent work suggests that uncorrelated Gaussian noise works just as well. TD3 paper (https://arxiv.org/pdf/1802.09477.pdf): "Unlike the original DDPG, we used uncorrelated noise for exploration as we found noise drawn from the Ornstein-Uhlenbeck (Uhlenbeck & Ornstein, 1930) process offered no performance benefits." D4PG paper (https://arxiv.org/pdf/1804.08617.pdf): "We experimented with correlated noise drawn from an Ornstein-Uhlenbeck process, as suggested by (Lillicrap et al., 2016), however we found this was unnecessary and did not add to performance."  
  
  ---
  
  ## 0 answers
  
  