Body: <p>I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all rewards are negative) I tested is of type <span class="math-container">$R(s, a, s')$</span>. During training, it can happen that not only one but several actions are applied simulataneously to the environement before a reward is returned: </p>
  
  <p><span class="math-container">$s_t →a_{t,1}, a_{t,2}, a_{t,3} →s_{t+1}$</span>   instead of   <span class="math-container">$s_t →a_t→s_{t+1}$</span>.</p>
  
  <p>Out of all actions applied, only one is generated by the agent. The others are either a copy of the agent's action or are new values. </p>
  
  <p>If I look at the tensorboard output of the trained agent, it looks rather horrific as displayed below (~ zero explained variance, key trainig values do not converge or behave weirdly, etc. etc.). </p>
  
  <p>Obviously, the training did not really work. Now I wonder what the reason for that is.</p>
  
  <ol>
  <li>Is it possible to train an agent using a reward function of type <span class="math-container">$R(s, a, s')$</span> even if several actions are applied simulataneously or is this not possible at all? Other agents I trained using a reward function of type <span class="math-container">$R(s,a)$</span> have a better tensorboard output so I guess that this is the problem.</li>
  <li>Or is maybe another reason more likely to be the root of the problem?  Like a bad observation space formulation or hyperparameter selection (both for RL algorithm and reward function used).</li>
  </ol>
  
  <p>Thanks for your help!</p>
  
  <p><a href="https://i.sstatic.net/2Sr9f.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2Sr9f.png" alt="enter image description here"></a></p>
  
  <p><a href="https://i.sstatic.net/LstdT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/LstdT.png" alt="enter image description here"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-07T14:26:38.923Z
Comments.Text: You could always change things to satisfy the usual convention $s_t→a_t→s_{t+1}$. You could either have an action vector $\mathbf a_t=(a_{t1},a_{t2},a_{t3})$ that includes all intermediate actions and then at the output have nodes for all possible action triplets or you could introduce intermediate states  $s_t→a_{t1}→s_{t1}→a_{t2}→s_{t2}→a_{t3}→s_{t+1}$. Also what does it mean that reward function is formulated as a loss function ?
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-07T15:10:04.687Z
Comments.Text: Hey, thanks for your answer. The suggestion using intermediate steps sounds quite interesting, however, it probably is not applicable to my environment since it requires all steps to be performed simultaneously. Or at least the reward function requires it. But I'll look some more into it.
  To specifiy the actions applied: only one of them is generated by the agent. The others are not.
  With the term loss function I want to imply that all rewards are negative.
Comments.UserDisplayName: mglss
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-07T15:33:33.287Z
Comments.Text: I think the term you're looking for is that reward function is negative definite function, meaning, for all function inputs, output is negative. As for the actions, the fact that only one action is generated by the agent means that those other actions shouldn't be treated as actions but rather state features. Consider this sequence: $s_t→a_{t,1}, a_{t,2}, a_{t,3} →s_{t+1}→a_{t+1,1}...$. Let's say actions $a_{t,1}$ and $a_{t+1,1}$ are generated by the agent. Then, the state features for generating action $a_{t+1,1}$ should be $\mathbf s'_{t+1}=(a_{t,2}, a_{t,3},s_{t+1})$
Comments.UserDisplayName: Brale
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T17:24:14.423Z
Comments.Text: Thanks for your input!. It helped me a lot finding out about how to solve my problem. In the end, the problem in my specific case was the formulation of the reward function and the corresponding state space formlation.
Comments.UserDisplayName: mglss
Tags: <reinforcement-learning><objective-functions><rewards><proximal-policy-optimization><gym>
Title: Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?
UserDisplayName: mglss
ThreadText: # Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?
  
  (Asked by: username_0 on 2019-08-07)
  
  I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all rewards are negative) I tested is of type $R(s, a, s')$. During training, it can happen that not only one but several actions are applied simulataneously to the environement before a reward is returned:
  
  $s\_t →a\_{t,1}, a\_{t,2}, a\_{t,3} →s\_{t+1}$ instead of $s\_t →a\_t→s\_{t+1}$.
  
  Out of all actions applied, only one is generated by the agent. The others are either a copy of the agent's action or are new values.
  
  If I look at the tensorboard output of the trained agent, it looks rather horrific as displayed below (~ zero explained variance, key trainig values do not converge or behave weirdly, etc. etc.).
  
  Obviously, the training did not really work. Now I wonder what the reason for that is.
  
  1. Is it possible to train an agent using a reward function of type $R(s, a, s')$ even if several actions are applied simulataneously or is this not possible at all? Other agents I trained using a reward function of type $R(s,a)$ have a better tensorboard output so I guess that this is the problem.
  2. Or is maybe another reason more likely to be the root of the problem? Like a bad observation space formulation or hyperparameter selection (both for RL algorithm and reward function used).
  
  Thanks for your help!
  
  [![enter image description here](https://i.sstatic.net/2Sr9f.png)](https://i.sstatic.net/2Sr9f.png)
  
  [![enter image description here](https://i.sstatic.net/LstdT.png)](https://i.sstatic.net/LstdT.png)
  
  **Tags:** reinforcement-learning, objective-functions, rewards, proximal-policy-optimization, gym
  
  > **Comments:**
  > **username_1**: You could always change things to satisfy the usual convention $s_t→a_t→s_{t+1}$. You could either have an action vector $\mathbf a_t=(a_{t1},a_{t2},a_{t3})$ that includes all intermediate actions and then at the output have nodes for all possible action triplets or you could introduce intermediate states  $s_t→a_{t1}→s_{t1}→a_{t2}→s_{t2}→a_{t3}→s_{t+1}$. Also what does it mean that reward function is formulated as a loss function ?  
  > **username_0**: Hey, thanks for your answer. The suggestion using intermediate steps sounds quite interesting, however, it probably is not applicable to my environment since it requires all steps to be performed simultaneously. Or at least the reward function requires it. But I'll look some more into it. To specifiy the actions applied: only one of them is generated by the agent. The others are not. With the term loss function I want to imply that all rewards are negative.  
  > **username_1**: I think the term you're looking for is that reward function is negative definite function, meaning, for all function inputs, output is negative. As for the actions, the fact that only one action is generated by the agent means that those other actions shouldn't be treated as actions but rather state features. Consider this sequence: $s_t→a_{t,1}, a_{t,2}, a_{t,3} →s_{t+1}→a_{t+1,1}...$. Let's say actions $a_{t,1}$ and $a_{t+1,1}$ are generated by the agent. Then, the state features for generating action $a_{t+1,1}$ should be $\mathbf s'_{t+1}=(a_{t,2}, a_{t,3},s_{t+1})$  
  > **username_0**: Thanks for your input!. It helped me a lot finding out about how to solve my problem. In the end, the problem in my specific case was the formulation of the reward function and the corresponding state space formlation.  
  
  ---
  
  ## 0 answers
  
  