Answers.Body: <p>It has an obvious answer: Network is conditioned to use tanh activation. Hence the action values in the buffer should be in the range [-1, 1], or unscaled values before action execution. As I am not using openai gym or other baselines for my learning, I overlooked this detail.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-26T20:54:46.327Z
Answers.Text: Can you give some details: "the buffer should be in [-1, 1] what do u mean
Answers.UserDisplayName: Rémy Hosseinkhan Boucher
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-27T08:14:29.497Z
Answers.Text: @rémy-hosseinkhan-boucher that is the action value range in buffer
Answers.UserDisplayName: goldfinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-16T08:50:42.497Z
Answers.LastActivityDate: 2022-07-27T08:12:09.707Z
Answers.LastEditDate: 2022-07-27T08:12:09.707Z
Answers.UserDisplayName: goldfinch
Body: <p>I have an environment where the agent action is in range [0, 1.57]. My actor network in DDPG has a tanh activation, and so the network values are in the range [-1,1]. Hence I change the scaling from [-1,1] to [0, 1.57] before an action is performed. My question is, when we store the transition <span class="math-container">$(s_t, a_t, r_t, s_{t+1})$</span> in replay buffer <span class="math-container">$R$</span>, should <span class="math-container">$a_t$</span> be in [-1,1] or [0, 0.157]</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-25T13:27:56.610Z
Comments.Text: I think it would make more sense to have your network use a sigmoid activation and then multiply the action by 1.57 as part of the network, and then everything should work out.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-26T20:45:21.423Z
Comments.Text: @DavidIreland tanh is used in order to have an action distribution (policy) possibly centered around zero.
Comments.UserDisplayName: Rémy Hosseinkhan Boucher
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-26T21:00:02.467Z
Comments.Text: @RémyHosseinkhanBoucher yes, I am fully aware. Here, OP doesn’t have a symmetric action space so it would make more sense to use sigmoid as this outputs in $(0, 1)$ and you can multiply the output by $x$ to get an action in the range $(0, x)$
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-27T13:26:40.877Z
Comments.Text: @DavidIreland ok sorry:)
Comments.UserDisplayName: Rémy Hosseinkhan Boucher
Tags: <reinforcement-learning><deep-rl><actor-critic-methods><ddpg>
Title: Replay buffer action range in DDPG
UserDisplayName: goldfinch
ThreadText: # Replay buffer action range in DDPG
  
  (Asked by: username_0 on 2022-05-24)
  
  I have an environment where the agent action is in range [0, 1.57]. My actor network in DDPG has a tanh activation, and so the network values are in the range [-1,1]. Hence I change the scaling from [-1,1] to [0, 1.57] before an action is performed. My question is, when we store the transition $(s\_t, a\_t, r\_t, s\_{t+1})$ in replay buffer $R$, should $a\_t$ be in [-1,1] or [0, 0.157]
  
  **Tags:** reinforcement-learning, deep-rl, actor-critic-methods, ddpg
  > **Comments:**
  > **username_1**: I think it would make more sense to have your network use a sigmoid activation and then multiply the action by 1.57 as part of the network, and then everything should work out.  
  > **username_2**: @DavidIreland tanh is used in order to have an action distribution (policy) possibly centered around zero.  
  > **username_1**: @RémyHosseinkhanBoucher yes, I am fully aware. Here, OP doesn’t have a symmetric action space so it would make more sense to use sigmoid as this outputs in $(0, 1)$ and you can multiply the output by $x$ to get an action in the range $(0, x)$  
  > **username_2**: @DavidIreland ok sorry:)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-06-16. Score: 1
  
  It has an obvious answer: Network is conditioned to use tanh activation. Hence the action values in the buffer should be in the range [-1, 1], or unscaled values before action execution. As I am not using openai gym or other baselines for my learning, I overlooked this detail.
  
  > **Comments:**
  > **username_2**: Can you give some details: "the buffer should be in [-1, 1] what do u mean  
  > **username_0**: @rémy-hosseinkhan-boucher that is the action value range in buffer  
  
  ---
  
  