Body: <p>I have a situation where an agent can take actions to enter a state where rewards can be obtained (or costs can be incurred), but the actions themselves have a small cost. In my reward function, I give the negative rewards immediately when the agent takes the action.</p>
  <p>As a result, if the agent takes these actions extremely frequently, the small, repeated costs of the actions outweigh the rewards gained during the very short periods between actions, and the agent does poorly. For some reason, my agent doesn't seem to be able to understand this. It acts erratically, taking actions at extremely high frequency.</p>
  <p>I have tried giving it a sense of &quot;frequency&quot; by having a state variable that increments by 0.02 for every time step that the agent is in the state where it can earn rewards, therefore giving it a direct measure of &quot;how long I've been in this state&quot;. This did not remove the erratic behavior.</p>
  <p>Is there some other way I can give it a sense of &quot;frequency&quot;? Why is the agent unable to grasp that it shouldn't keep &quot;flip-flopping&quot; and incurring these costs?</p>
  <p>EDIT: From @NeilSlater's comment, adding more info.</p>
  <ol>
  <li>Yes, the agent can just &quot;wait&quot; with no associated cost. There are two possible states (one in which it just sits there and nothing happens, and the other in which it can earn rewards or incur costs, depending on various factors). There are 2 possible actions - the agent can either 'stay in current state' (meaning no costs/rewards in the &quot;inactive&quot; state, and costs/rewards in 'active' state), or it can move between states, which incurs a small cost.</li>
  <li>My environment is not naturally episodic. I chose a somewhat arbitrary episode length to have some end.</li>
  <li>I'm using a discount factor (gamma) on the order of 0.9 to 0.975 . Changing it within that range doesn't appear to make a huge difference (though I can investigate this more)</li>
  <li>I'm not very familiar with the technical definitions of MDP vs. POMDP, but given that there is a huge stochasticity in my observation space, I think POMDP might be correct. Nonetheless, the costs associated with moving between states are completely consistent, so I don't know how to understand this behavior.</li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-26T21:00:55.823Z
Comments.Text: Could you clarify: Is one of the agent's action choices "no action" or "wait" with no associated cost? Also worth stating if your environment is episodic vs continuing, and what discount factor you are using (or are you using average reward). My biggest suspicion though is that you have some important hidden state - but I cannot tell anything about whether your state model is correct or if you might have a POMDP from your description.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-27T03:19:34.527Z
Comments.Text: @NeilSlater I'm sorry for not being clear originally. Thank you for the questions, I added all answers in my edit.
Comments.UserDisplayName: Vladimir Belik
Tags: <deep-rl><reward-design>
Title: How to teach DRL agent to avoid erratic, high frequency behavior (when costs for taking actions are already included)?
UserDisplayName: Vladimir Belik
ThreadText: # How to teach DRL agent to avoid erratic, high frequency behavior (when costs for taking actions are already included)?
  
  (Asked by: username_0 on 2022-06-26)
  
  I have a situation where an agent can take actions to enter a state where rewards can be obtained (or costs can be incurred), but the actions themselves have a small cost. In my reward function, I give the negative rewards immediately when the agent takes the action.
  
  As a result, if the agent takes these actions extremely frequently, the small, repeated costs of the actions outweigh the rewards gained during the very short periods between actions, and the agent does poorly. For some reason, my agent doesn't seem to be able to understand this. It acts erratically, taking actions at extremely high frequency.
  
  I have tried giving it a sense of "frequency" by having a state variable that increments by 0.02 for every time step that the agent is in the state where it can earn rewards, therefore giving it a direct measure of "how long I've been in this state". This did not remove the erratic behavior.
  
  Is there some other way I can give it a sense of "frequency"? Why is the agent unable to grasp that it shouldn't keep "flip-flopping" and incurring these costs?
  
  EDIT: From @username_1's comment, adding more info.
  
  1. Yes, the agent can just "wait" with no associated cost. There are two possible states (one in which it just sits there and nothing happens, and the other in which it can earn rewards or incur costs, depending on various factors). There are 2 possible actions - the agent can either 'stay in current state' (meaning no costs/rewards in the "inactive" state, and costs/rewards in 'active' state), or it can move between states, which incurs a small cost.
  2. My environment is not naturally episodic. I chose a somewhat arbitrary episode length to have some end.
  3. I'm using a discount factor (gamma) on the order of 0.9 to 0.975 . Changing it within that range doesn't appear to make a huge difference (though I can investigate this more)
  4. I'm not very familiar with the technical definitions of MDP vs. POMDP, but given that there is a huge stochasticity in my observation space, I think POMDP might be correct. Nonetheless, the costs associated with moving between states are completely consistent, so I don't know how to understand this behavior.
  
  **Tags:** deep-rl, reward-design
  
  > **Comments:**
  > **username_1**: Could you clarify: Is one of the agent's action choices "no action" or "wait" with no associated cost? Also worth stating if your environment is episodic vs continuing, and what discount factor you are using (or are you using average reward). My biggest suspicion though is that you have some important hidden state - but I cannot tell anything about whether your state model is correct or if you might have a POMDP from your description.  
  > **username_0**: @username_1 I'm sorry for not being clear originally. Thank you for the questions, I added all answers in my edit.  
  
  ---
  
  ## 0 answers
  
  