Answers.Body: <p>I would recommend doing is allowing your network to output any real number and then clipping the output. For instance, I was working with an agent that had to learn an angle between <span class="math-container">$[0, 2\pi]$</span> and <span class="math-container">$[0, 1]$</span>. If the network outputted e.g. 10 in the first dimension then this would just be clipped to <span class="math-container">$2\pi$</span>.</p>
  <p>This way the agent only learns about actions within the action space and the weights of the network would eventually be adjusted to only output actions within this action space, provided that the boundaries aren't the optimal actions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-08T10:50:44.210Z
Answers.LastActivityDate: 2021-01-08T10:50:44.210Z
Answers.UserDisplayName: David
Body: <p>I am using DDPG to solve a RL problem. The action space is given by the Cartesian product <span class="math-container">$[0,20]^4\times[0,6]^4$</span>. The <code>actor</code> is implemented as a deep neural network with an output dimension equals to <span class="math-container">$8$</span> with <code>tanh</code> activation.</p>
  <p>So, given a state <code>s</code>, an action is given by <code>a = actor(s)</code> where <code>a</code> contains real numbers in <code>[-1,1]</code>. Next, I map this action <code>a</code> into a valid action <code>valid_a</code> that belongs to the action space <span class="math-container">$[0,20]^4\times[0,6]^4$</span>. Than, I use <code>valid_a</code> to calculate the reward.</p>
  <p>My question is: how does the DDPG algorithm know about this mapping that I am doing? In what part of the DDPG algorithm should I specify this mapping? Should I provide a bijective mapping to guarantee that the DDPG algorithm learns bad from good actions?</p>
  
Tags: <reinforcement-learning><deep-rl><ddpg><mapping-space>
Title: How does DDPG algorithm know about my action mapping function?
UserDisplayName: zdm
ThreadText: # How does DDPG algorithm know about my action mapping function?
  
  (Asked by: username_0 on 2021-01-08)
  
  I am using DDPG to solve a RL problem. The action space is given by the Cartesian product $[0,20]^4\times[0,6]^4$. The `actor` is implemented as a deep neural network with an output dimension equals to $8$ with `tanh` activation.
  
  So, given a state `s`, an action is given by `a = actor(s)` where `a` contains real numbers in `[-1,1]`. Next, I map this action `a` into a valid action `valid_a` that belongs to the action space $[0,20]^4\times[0,6]^4$. Than, I use `valid_a` to calculate the reward.
  
  My question is: how does the DDPG algorithm know about this mapping that I am doing? In what part of the DDPG algorithm should I specify this mapping? Should I provide a bijective mapping to guarantee that the DDPG algorithm learns bad from good actions?
  
  **Tags:** reinforcement-learning, deep-rl, ddpg, mapping-space
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-01-08. Score: 1 | Accepted answer
  
  I would recommend doing is allowing your network to output any real number and then clipping the output. For instance, I was working with an agent that had to learn an angle between $[0, 2\pi]$ and $[0, 1]$. If the network outputted e.g. 10 in the first dimension then this would just be clipped to $2\pi$.
  
  This way the agent only learns about actions within the action space and the weights of the network would eventually be adjusted to only output actions within this action space, provided that the boundaries aren't the optimal actions.
  
  
  ---
  
  