Body: <p>I have a simple question about model-free reinforcement. In a model I'm writing about, I want to know the value 'gain' we'd get for executing an action, relative to the current state. That is, what will I get if I moved from the current state <span class="math-container">$s$</span> taking action <span class="math-container">$a$</span>.</p>
  
  <p>The measure I want is:</p>
  
  <p><span class="math-container">$$G(s, a)=V(s^{\prime})-V(s)$$</span></p>
  
  <p>where <span class="math-container">$s'$</span> is the state that I would transition to if the underlying MDP was deterministic. If the MDP has a stochastic transition function, the model I want is:</p>
  
  <p><span class="math-container">$$G(s, a)=\left[\sum_{s' \in S } P(s^{\prime} \mid a, s) V(s^{\prime})\right]-V(s)$$</span></p>
  
  <p>In a model-free environment, we don't have <span class="math-container">$P(s' \mid a,s)$</span>.</p>
  
  <p>If we had a Q-function <span class="math-container">$Q(s,a)$</span>, could we represent <span class="math-container">$G(s,a)$</span>?</p>
  
  <p>NOTE: This is not the same as an 'advantage function' as first proposed by Baird (Leemon C Baird. Reinforcement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks, pages 448–2453. IEEE, 1994.), which means the advantage of actions relative to the optimal action. What I'm looking for is the gain of actions relative to the current state.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-02T07:22:49.503Z
Comments.Text: It would be a good idea to call your value something other than "advantage" and to use a different notation than $A(s,a)$. Your question will otherwise be confusing to any reader familiar with the normal use of the term (which you refer), which is well known in reinforcement learning. I suggest you change it using [edit]
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-02T11:36:02.977Z
Comments.Text: I suggest you also don't use the letter $G$, given that it's used for the return. Use the letter $B$ or any less common letter in RL.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-02T22:19:42.393Z
Comments.Text: B(s) is the baseline function in actor-critic, which is arguably more confusing than G. To me G(s,a) is fine because it is easily distinguished between G_t or G.
Comments.UserDisplayName: gibberblot
Tags: <reinforcement-learning>
Title: Calculating the advantage 'gain' of actions in model-free reinforcement learning
UserDisplayName: gibberblot
ThreadText: # Calculating the advantage 'gain' of actions in model-free reinforcement learning
  
  (Asked by: username_0 on 2020-06-01)
  
  I have a simple question about model-free reinforcement. In a model I'm writing about, I want to know the value 'gain' we'd get for executing an action, relative to the current state. That is, what will I get if I moved from the current state $s$ taking action $a$.
  
  The measure I want is:
  
  $$G(s, a)=V(s^{\prime})-V(s)$$
  
  where $s'$ is the state that I would transition to if the underlying MDP was deterministic. If the MDP has a stochastic transition function, the model I want is:
  
  $$G(s, a)=\left[\sum\_{s' \in S } P(s^{\prime} \mid a, s) V(s^{\prime})\right]-V(s)$$
  
  In a model-free environment, we don't have $P(s' \mid a,s)$.
  
  If we had a Q-function $Q(s,a)$, could we represent $G(s,a)$?
  
  NOTE: This is not the same as an 'advantage function' as first proposed by Baird (Leemon C Baird. Reinforcement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks, pages 448–2453. IEEE, 1994.), which means the advantage of actions relative to the optimal action. What I'm looking for is the gain of actions relative to the current state.
  
  **Tags:** reinforcement-learning
  
  > **Comments:**
  > **username_1**: It would be a good idea to call your value something other than "advantage" and to use a different notation than $A(s,a)$. Your question will otherwise be confusing to any reader familiar with the normal use of the term (which you refer), which is well known in reinforcement learning. I suggest you change it using [edit]  
  > **username_2**: I suggest you also don't use the letter $G$, given that it's used for the return. Use the letter $B$ or any less common letter in RL.  
  > **username_0**: B(s) is the baseline function in actor-critic, which is arguably more confusing than G. To me G(s,a) is fine because it is easily distinguished between G_t or G.  
  
  ---
  
  ## 0 answers
  
  