Answers.Body: <blockquote>
  <p>Should I make each step an episode so the action will be always applied at each step ? (i.e., should I consider a step the transition between the initial position of the balls and the their final position?).</p>
  </blockquote>
  <p>Yes from the perspective of the <em>agent</em> this is a single-step problem. As such, it is not a full reinforcement learning problem, but can also be considered a contextual bandit problem.</p>
  <p>The time steps of the <em>simulation</em> are not relevant to the agent if it does not get to take actions for them. They are external to the agent, and an implementation detail of how you calculate the outcome of its action.</p>
  <p>You <em>can</em> still treat this as a full RL problem where the agent has one available &quot;do nothing&quot; action for later steps. You might want to do that in case you would like the agent code to predict the outcome as it observes later states. This <em>may</em> even improve the performance of the agent, as all those observations will allow it to learn physics rules that it might be able to generalise from when considering a starting state. For this to work, you may need to encode more details in the state though, because a static snapshot of positions is not enough data to predict what will happen next.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-18T12:40:31.107Z
Answers.Text: Does using a "do nothing" action result in sparse rewards, potentially slowing learning?
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-18T14:43:54.010Z
Answers.Text: @LucaAnzalone that depends entirely on how rewards are calculated, and you have to balance any negative impact from that against the possibilities of easier learning of the physics. I don't think it's easy to predict without trying it
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-16T13:56:53.347Z
Answers.LastActivityDate: 2023-06-16T13:56:53.347Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am working in an reinforcement learning environment with 1-dimensional action space. My action is only used at the first timestep of an episode and never again. In other word the action only affects the agent's behavior at timestep 1 and is not used at any of the later timesteps.</p>
  <p>To illustrate my specific case, here is an image of my enviroment:</p>
  <p><a href="https://i.sstatic.net/R70ZU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/R70ZU.png" alt="enter image description here" /></a></p>
  <p>At step 0 red ball recives the action (force at a certain angle) and starts moving. At each timestep the coordinates of the red and blue ball are computed, but we cannot interfere with the enviroment anymore (the final configuration is already determined by the initail impulse and the physics).</p>
  <p>Should I make each step an episode so the action will be always applied at each step ? (i.e., should I consider a step the transition between the initial position of the balls and the their final position?).</p>
  <p>I am new to reinforcement learning so I apologize if this is a silly question</p>
  
Tags: <reinforcement-learning><deep-rl><action-spaces>
Title: Take action only at the beginning of the episode, not during each step
UserDisplayName: Optical_flow_lover
ThreadText: # Take action only at the beginning of the episode, not during each step
  
  (Asked by: username_0 on 2023-06-16)
  
  I am working in an reinforcement learning environment with 1-dimensional action space. My action is only used at the first timestep of an episode and never again. In other word the action only affects the agent's behavior at timestep 1 and is not used at any of the later timesteps.
  
  To illustrate my specific case, here is an image of my enviroment:
  
  [![enter image description here](https://i.sstatic.net/R70ZU.png)](https://i.sstatic.net/R70ZU.png)
  
  At step 0 red ball recives the action (force at a certain angle) and starts moving. At each timestep the coordinates of the red and blue ball are computed, but we cannot interfere with the enviroment anymore (the final configuration is already determined by the initail impulse and the physics).
  
  Should I make each step an episode so the action will be always applied at each step ? (i.e., should I consider a step the transition between the initial position of the balls and the their final position?).
  
  I am new to reinforcement learning so I apologize if this is a silly question
  
  **Tags:** reinforcement-learning, deep-rl, action-spaces
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-16. Score: 4
  
  > Should I make each step an episode so the action will be always applied at each step ? (i.e., should I consider a step the transition between the initial position of the balls and the their final position?).
  
  Yes from the perspective of the *agent* this is a single-step problem. As such, it is not a full reinforcement learning problem, but can also be considered a contextual bandit problem.
  
  The time steps of the *simulation* are not relevant to the agent if it does not get to take actions for them. They are external to the agent, and an implementation detail of how you calculate the outcome of its action.
  
  You *can* still treat this as a full RL problem where the agent has one available "do nothing" action for later steps. You might want to do that in case you would like the agent code to predict the outcome as it observes later states. This *may* even improve the performance of the agent, as all those observations will allow it to learn physics rules that it might be able to generalise from when considering a starting state. For this to work, you may need to encode more details in the state though, because a static snapshot of positions is not enough data to predict what will happen next.
  
  > **Comments:**
  > **username_2**: Does using a "do nothing" action result in sparse rewards, potentially slowing learning?  
  > **username_1**: @LucaAnzalone that depends entirely on how rewards are calculated, and you have to balance any negative impact from that against the possibilities of easier learning of the physics. I don't think it's easy to predict without trying it  
  
  ---
  
  