Answers.Body: <p>You should be able to learn a good policy even if you use the first two actions only at the first timestep.</p>
  <p>Using <a href="https://spinningup.openai.com/en/latest/algorithms/sac.html" rel="nofollow noreferrer">this OpenAI reference</a>, the loss for the state action value function (from which the policy loss is later derived) is:</p>
  <p><span class="math-container">$$L(\phi) = \mathbb{E}_{(s, a, r, s') \sim D}\left[\left(Q(s,a|\phi) - (r + \gamma Q(s', a'|\phi_{target})\right)^2\right]$$</span></p>
  <p>where <span class="math-container">$D$</span> is a set of transitions, <span class="math-container">$\phi_{target}$</span> are &quot;old&quot; parameters for the action state value function which are left unchanged in the parameter update, and <span class="math-container">$a' \sim \pi(.|s,\theta)$</span>.</p>
  <p>Note that I've simplified the equation for clarity.</p>
  <p>The expectation in the loss is replaced in the actual algorithm with an average on a batch of transitions.</p>
  <p>At timestep 0, the target <span class="math-container">$r + \gamma Q(s', a'|\phi_{target})$</span> for <span class="math-container">$Q(s_{t_0},a_{t_0}|\phi)$</span> (with <span class="math-container">$a' \sim \pi(.|a_{t_0}, 
  \theta)$</span>) in the loss will be non-zero, because <span class="math-container">$Q(s', a'|\phi_{target})$</span> will be non-zero and will reflect the value of <span class="math-container">$(s',a')$</span> accurately (e.g. thanks to transitions which happen at later timesteps).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-04T12:56:01.847Z
Answers.Text: Thank you for your answer. But, I believe I did not make it clear in my question (I have edited it now). I am getting rewards at each timestep but, only use the third action at every timestep and not the first two, which are used only on the first timestep of every episode. My concern is that since the first two actions do not affect the agent's behavior at each timestep, the RL policy might not be good at choosing good values for $a_1$ and $a_2$.
Answers.UserDisplayName: user13018205
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-08T18:43:08.467Z
Answers.Text: I had understood that the first two actions are only used in the first timestep. When I say "even if you don't see a reward", that wasn't to say that I thought your issue was with having a reward or not, but that even if you didn't see a reward in the first timestep, the policy would be accurate for the action $(a_1, a_2)$ because the target will be non-zero in either case (reward or not). I've amended the answer to be clearer
Answers.UserDisplayName: Raphael Lopez Kaufman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-02T20:30:33.473Z
Answers.LastActivityDate: 2022-08-08T18:49:45.343Z
Answers.LastEditDate: 2022-08-08T18:49:45.343Z
Answers.UserDisplayName: Raphael Lopez Kaufman
Body: <p>I am working in an environment with 3-dimensional action space. The first two actions are only used at the first timestep and never again. The third action is used at every timestep.</p>
  <p>Say, the action is <span class="math-container">$a = (a_1, a_2, a_3)$</span>. At the start of an episode <span class="math-container">$i$</span>, the agent uses actions <span class="math-container">$a_1, a_2$</span> only at timestep 1. Action <span class="math-container">$a_3$</span> is used at every timestep in the episode starting from 1 till the horizon H. The agent receives rewards <span class="math-container">$r_i$</span> at each timestep <span class="math-container">$i$</span> till the end of the episode.</p>
  <p>I am using SAC. Since the actions <span class="math-container">$a_1, a_2$</span> only affect the agent's behavior at timestep 1 and are not used at any of the later timesteps, I am not sure if the RL policy will get better at choosing &quot;good&quot; values for <span class="math-container">$a_1, a_2$</span>.</p>
  <p>Will the RL be able to learn a good policy even if it doesn't quite see the effects of the first two actions in the episode data except at the first timestep?</p>
  
Tags: <reinforcement-learning><deep-rl><soft-actor-critic>
Title: Can RL still learn if part of my actions are only used once, at the beginning of the episode?
UserDisplayName: user13018205
ThreadText: # Can RL still learn if part of my actions are only used once, at the beginning of the episode?
  
  (Asked by: username_0 on 2022-08-02)
  
  I am working in an environment with 3-dimensional action space. The first two actions are only used at the first timestep and never again. The third action is used at every timestep.
  
  Say, the action is $a = (a\_1, a\_2, a\_3)$. At the start of an episode $i$, the agent uses actions $a\_1, a\_2$ only at timestep 1. Action $a\_3$ is used at every timestep in the episode starting from 1 till the horizon H. The agent receives rewards $r\_i$ at each timestep $i$ till the end of the episode.
  
  I am using SAC. Since the actions $a\_1, a\_2$ only affect the agent's behavior at timestep 1 and are not used at any of the later timesteps, I am not sure if the RL policy will get better at choosing "good" values for $a\_1, a\_2$.
  
  Will the RL be able to learn a good policy even if it doesn't quite see the effects of the first two actions in the episode data except at the first timestep?
  
  **Tags:** reinforcement-learning, deep-rl, soft-actor-critic
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-08-02. Score: 0 | Accepted answer
  
  You should be able to learn a good policy even if you use the first two actions only at the first timestep.
  
  Using [this OpenAI reference](https://spinningup.openai.com/en/latest/algorithms/sac.html), the loss for the state action value function (from which the policy loss is later derived) is:
  
  $$L(\phi) = \mathbb{E}\_{(s, a, r, s') \sim D}\left[\left(Q(s,a|\phi) - (r + \gamma Q(s', a'|\phi\_{target})\right)^2\right]$$
  
  where $D$ is a set of transitions, $\phi\_{target}$ are "old" parameters for the action state value function which are left unchanged in the parameter update, and $a' \sim \pi(.|s,\theta)$.
  
  Note that I've simplified the equation for clarity.
  
  The expectation in the loss is replaced in the actual algorithm with an average on a batch of transitions.
  
  At timestep 0, the target $r + \gamma Q(s', a'|\phi\_{target})$ for $Q(s\_{t\_0},a\_{t\_0}|\phi)$ (with $a' \sim \pi(.|a\_{t\_0},
  \theta)$) in the loss will be non-zero, because $Q(s', a'|\phi\_{target})$ will be non-zero and will reflect the value of $(s',a')$ accurately (e.g. thanks to transitions which happen at later timesteps).
  
  > **Comments:**
  > **username_0**: Thank you for your answer. But, I believe I did not make it clear in my question (I have edited it now). I am getting rewards at each timestep but, only use the third action at every timestep and not the first two, which are used only on the first timestep of every episode. My concern is that since the first two actions do not affect the agent's behavior at each timestep, the RL policy might not be good at choosing good values for $a_1$ and $a_2$.  
  > **username_1**: I had understood that the first two actions are only used in the first timestep. When I say "even if you don't see a reward", that wasn't to say that I thought your issue was with having a reward or not, but that even if you didn't see a reward in the first timestep, the policy would be accurate for the action $(a_1, a_2)$ because the target will be non-zero in either case (reward or not). I've amended the answer to be clearer  
  
  ---
  
  