Answers.Body: <p>A primary reason to repeat an action of a series of timesteps is that your environment may require more than one timestep to process the timestep.  Said another way, changing the action every timestep might prove to hamper the overall learning process significantly.</p>
  <p>As an example, consider using RL with something from OpenAI Gym.  Imagine that you are trying to train an agent to play something like Mario Brothers.  If your agent takes a different random action at every timestep, how long might it take the agent to realize that it needs to keep sending &quot;Right&quot; to improve the x position?  It might be better to configure your agent such that each action is repeated for, say, four timesteps, allowing the action to begin to take on real meaning in the environment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-19T12:48:58.727Z
Answers.LastActivityDate: 2022-03-19T12:48:58.727Z
Answers.UserDisplayName: David Hoelzer
Body: <p>What's the benefit of repeating an action for a consecutive number of time steps? Is there a way to tell if an agent in a given environment might perform better from repeated actions?</p>
  <p>I came across an <code>action repeat</code> hyperparameter used in the experiments of <a href="https://arxiv.org/pdf/2007.04309.pdf" rel="nofollow noreferrer">this paper</a> (Table 11.) and was wondering what the objective of this hyperparameter might be. This happens during training where the action selected by a policy at step <span class="math-container">$t$</span> is executed for a repeat number of timesteps <span class="math-container">$t_r$</span> after which the policy is then queried for another action at the current state <span class="math-container">$s_{t + t_r}$</span>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-18T08:57:15.543Z
Comments.Text: You mean hold a given action selection for more than one timestep like _sticky action_?
Comments.UserDisplayName: HenDoNR
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-18T09:34:19.803Z
Comments.Text: I assume you're talking about behaviour policies, so during learning. It might a good idea to explicitly state this in your post, even though it seems obvious. I would also recommend that you explain why you're asking this question: just out of curiosity, or do you have something more specific that you can share with us?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-19T10:44:00.527Z
Comments.Text: Yes @HenDoNR, action selection for more than a signle timestep
Comments.UserDisplayName: mugoh
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-19T10:51:16.040Z
Comments.Text: I've edited the question to include the suggested changes @nbro
Comments.UserDisplayName: mugoh
Tags: <reinforcement-learning><policies><action-spaces>
Title: What's the benefit of repeating an action for a consecutive number of time steps?
UserDisplayName: mugoh
ThreadText: # What's the benefit of repeating an action for a consecutive number of time steps?
  
  (Asked by: username_0 on 2022-03-18)
  
  What's the benefit of repeating an action for a consecutive number of time steps? Is there a way to tell if an agent in a given environment might perform better from repeated actions?
  
  I came across an `action repeat` hyperparameter used in the experiments of [this paper](https://arxiv.org/pdf/2007.04309.pdf) (Table 11.) and was wondering what the objective of this hyperparameter might be. This happens during training where the action selected by a policy at step $t$ is executed for a repeat number of timesteps $t\_r$ after which the policy is then queried for another action at the current state $s\_{t + t\_r}$.
  
  **Tags:** reinforcement-learning, policies, action-spaces
  > **Comments:**
  > **username_1**: You mean hold a given action selection for more than one timestep like _sticky action_?  
  > **username_2**: I assume you're talking about behaviour policies, so during learning. It might a good idea to explicitly state this in your post, even though it seems obvious. I would also recommend that you explain why you're asking this question: just out of curiosity, or do you have something more specific that you can share with us?  
  > **username_0**: Yes @HenDoNR, action selection for more than a signle timestep  
  > **username_0**: I've edited the question to include the suggested changes @nbro  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-03-19. Score: 1 | Accepted answer
  
  A primary reason to repeat an action of a series of timesteps is that your environment may require more than one timestep to process the timestep. Said another way, changing the action every timestep might prove to hamper the overall learning process significantly.
  
  As an example, consider using RL with something from OpenAI Gym. Imagine that you are trying to train an agent to play something like Mario Brothers. If your agent takes a different random action at every timestep, how long might it take the agent to realize that it needs to keep sending "Right" to improve the x position? It might be better to configure your agent such that each action is repeated for, say, four timesteps, allowing the action to begin to take on real meaning in the environment.
  
  
  ---
  
  