Answers.Body: <p>As said, you will have to include previous input state(s) in your training input patterns.</p>
  <p>Suppose we use straightforward NN backpropagation learning..</p>
  <p>You would expand the input layer with <em>additional neurons and weights</em> connected to the past. The time window neurons should introduce <em>additional</em> weights into the first hidden layer. A neural net architecture doing this is called Time Delay neural net (TDNN)</p>
  <p><a href="https://stats.stackexchange.com/questions/160070/difference-between-time-delayed-neural-networks-and-recurrent-neural-networks">https://stats.stackexchange.com/questions/160070/difference-between-time-delayed-neural-networks-and-recurrent-neural-networks</a></p>
  <p>I used TDNN in the past for signal processing. My TDNN includes the past of N inputs, supporting M steps in the past. A flexible TDNN can also be configured to connect some hidden layer past output to the next hidden layer.</p>
  <p><a href="https://neuron.eng.wayne.edu/tarek/MITbook/chap5/5_4.html" rel="nofollow noreferrer">https://neuron.eng.wayne.edu/tarek/MITbook/chap5/5_4.html</a></p>
  <p><a href="https://www.mathworks.com/help/deeplearning/ref/timedelaynet.html" rel="nofollow noreferrer">https://www.mathworks.com/help/deeplearning/ref/timedelaynet.html</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-26T16:14:09.857Z
Answers.LastActivityDate: 2021-01-26T16:23:57.660Z
Answers.LastEditDate: 2021-01-26T16:23:57.660Z
Answers.UserDisplayName: Goodies
Answers.Body: <p>The answer to both your concerns is:</p>
  <ul>
  <li>Add the previous action choice to the state representation.</li>
  </ul>
  <p>It is all you need to do. It gives the agent the data it needs to learn the association of negative reward from not matching the previous action.</p>
  <p>By making this data part of the state, you re-establish the <a href="https://en.wikipedia.org/wiki/Markov_property" rel="nofollow noreferrer">Markov property</a> in the MDP model of the environment, which you had otherwise lost by making the reward dependent on a variable that was both systematically changing and hidden from the agent.</p>
  <p>The state in a MDP is often not just the current observations that the environment provides, but can include any relevant knowledge that the agent has. At the extreme that can include a complete history of all observations and actions taken to date. It is common practice to derive the state as a summary of recent history of observations and actions taken so far. In your case, all you need do is concatenate the previous action to the observation, because you know about the constraint and how it affects optimisation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-26T10:18:08.467Z
Answers.Text: Thanks! That was my initial idea too. You are absolutely right, that MDP property has to be restored. My concern is that adding previous action to the state representation might be not enough, as it will take some time for the agent to learn to associate this state feature with reward. Maybe there is a more explicit way of influencing learning that switching actions bears a penalty?
Answers.UserDisplayName: FQT
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-26T10:25:28.770Z
Answers.Text: @FQT: There is no specific clever way that is normally used in RL to solve this issue as far as I know. However, the new MDP with the previous action *might* be solved faster using a different algorithm, as that part of the behaviour is deterministic. You could try to re-structure the agent in other ways too, because you know specific things aboutthis penalty that you might not know about the rest of the environment (for instance reward prediction models could focus only on the other part of the environment with action matching hard-coded), but modifying state is simpler and IMO best.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-26T09:41:27.293Z
Answers.LastActivityDate: 2021-01-26T10:18:36.167Z
Answers.LastEditDate: 2021-01-26T10:18:36.167Z
Answers.UserDisplayName: Neil Slater
Body: <p><strong>Problem description:</strong></p>
  <p>Suppose we have an environment, where a reward at time step <span class="math-container">$t$</span> is dependent not only on the current action, but also on previous action in the following way:</p>
  <ul>
  <li>if current action == previous action, you get reward = <span class="math-container">$R(a,s)$</span></li>
  <li>if current action != previous action, you get reward = <span class="math-container">$R(a,s) - \text{penalty}$</span></li>
  </ul>
  <p>In this environment, switching actions bears a significant cost. We would like the RL algorithm to learn optimal actions under the constraint that switching action is costly, i.e. we would like to stay in selected action as long as possible.</p>
  <p>The penalty is significantly higher than an immediate reward, so if we do not take it into account, the model evaluation will have a negative total reward with almost 100% probability, since the agent will be constantly switching and extracting rewards from environment that are smaller than the cost of switching actions.</p>
  <p>Action space is small (2 actions: left, right). I'm trying to beat this game with PPO (Proximal Policy Optimization)</p>
  <p><strong>Questions</strong></p>
  <ul>
  <li><p>How one might address this constraint: i.e. explicitly make the agent learn that switching is costly and it's worth sitting in one action even if immediate rewards are negative?</p>
  </li>
  <li><p>How can you make the RL algorithm learn that it's not the reward term <span class="math-container">$R(a_t|s_t)$</span> that is negative, and thus decreasing <span class="math-container">$Q(a_t|s_t)$</span> and <span class="math-container">$V(s_t)$</span>, but it's the penalty term (taking the action that is different from the previous action at step <span class="math-container">$t-1$</span>) that is pushing total reward down?</p>
  </li>
  </ul>
  
Tags: <reinforcement-learning><proximal-policy-optimization><reward-design><state-spaces><markov-property>
Title: Reinforcement Learning algorithm with rewards dependent both on previous action and current action
UserDisplayName: FQT
ThreadText: # Reinforcement Learning algorithm with rewards dependent both on previous action and current action
  
  (Asked by: username_0 on 2021-01-26)
  
  **Problem description:**
  
  Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way:
  
  * if current action == previous action, you get reward = $R(a,s)$
  * if current action != previous action, you get reward = $R(a,s) - \text{penalty}$
  
  In this environment, switching actions bears a significant cost. We would like the RL algorithm to learn optimal actions under the constraint that switching action is costly, i.e. we would like to stay in selected action as long as possible.
  
  The penalty is significantly higher than an immediate reward, so if we do not take it into account, the model evaluation will have a negative total reward with almost 100% probability, since the agent will be constantly switching and extracting rewards from environment that are smaller than the cost of switching actions.
  
  Action space is small (2 actions: left, right). I'm trying to beat this game with PPO (Proximal Policy Optimization)
  
  **Questions**
  
  * How one might address this constraint: i.e. explicitly make the agent learn that switching is costly and it's worth sitting in one action even if immediate rewards are negative?
  * How can you make the RL algorithm learn that it's not the reward term $R(a\_t|s\_t)$ that is negative, and thus decreasing $Q(a\_t|s\_t)$ and $V(s\_t)$, but it's the penalty term (taking the action that is different from the previous action at step $t-1$) that is pushing total reward down?
  
  **Tags:** reinforcement-learning, proximal-policy-optimization, reward-design, state-spaces, markov-property
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-01-26. Score: 3 | Accepted answer
  
  The answer to both your concerns is:
  
  * Add the previous action choice to the state representation.
  
  It is all you need to do. It gives the agent the data it needs to learn the association of negative reward from not matching the previous action.
  
  By making this data part of the state, you re-establish the [Markov property](https://en.wikipedia.org/wiki/Markov_property) in the MDP model of the environment, which you had otherwise lost by making the reward dependent on a variable that was both systematically changing and hidden from the agent.
  
  The state in a MDP is often not just the current observations that the environment provides, but can include any relevant knowledge that the agent has. At the extreme that can include a complete history of all observations and actions taken to date. It is common practice to derive the state as a summary of recent history of observations and actions taken so far. In your case, all you need do is concatenate the previous action to the observation, because you know about the constraint and how it affects optimisation.
  
  > **Comments:**
  > **username_0**: Thanks! That was my initial idea too. You are absolutely right, that MDP property has to be restored. My concern is that adding previous action to the state representation might be not enough, as it will take some time for the agent to learn to associate this state feature with reward. Maybe there is a more explicit way of influencing learning that switching actions bears a penalty?  
  > **username_2**: @FQT: There is no specific clever way that is normally used in RL to solve this issue as far as I know. However, the new MDP with the previous action *might* be solved faster using a different algorithm, as that part of the behaviour is deterministic. You could try to re-structure the agent in other ways too, because you know specific things aboutthis penalty that you might not know about the rest of the environment (for instance reward prediction models could focus only on the other part of the environment with action matching hard-coded), but modifying state is simpler and IMO best.  
  
  ---
  
  ### Answer by username_1 on 2021-01-26. Score: 0
  
  As said, you will have to include previous input state(s) in your training input patterns.
  
  Suppose we use straightforward NN backpropagation learning..
  
  You would expand the input layer with *additional neurons and weights* connected to the past. The time window neurons should introduce *additional* weights into the first hidden layer. A neural net architecture doing this is called Time Delay neural net (TDNN)
  
  <https://stats.stackexchange.com/questions/160070/difference-between-time-delayed-neural-networks-and-recurrent-neural-networks>
  
  I used TDNN in the past for signal processing. My TDNN includes the past of N inputs, supporting M steps in the past. A flexible TDNN can also be configured to connect some hidden layer past output to the next hidden layer.
  
  <https://neuron.eng.wayne.edu/tarek/MITbook/chap5/5_4.html>
  
  <https://www.mathworks.com/help/deeplearning/ref/timedelaynet.html>
  
  
  ---
  
  