Answers.Body: <p>Your criteria #4:</p>
  <blockquote>
  <ol start="4">
  <li>It is possible to enumerate all next states <span class="math-container">$s'$</span> for a given state <span class="math-container">$s$</span>.</li>
  </ol>
  </blockquote>
  <p>makes me think you are describing <em>afterstate</em> representation.</p>
  <p>Yes, you can really ignore actions for model-free RL in that case, but it's not a matter of &quot;cancelling out&quot; actions starting from definitions in Q-learning. Instead, actions and action-value functions simply don't exist in this subset of reinforcement learning problems. The agent's &quot;action&quot; is to select the next state directly, there is no separate action space, and nothing need get formally labelled with a separate type of action (<span class="math-container">$a \in \mathcal{A}$</span>). This is common in board games for example, although these have different complications to deal with adversarial agents.</p>
  <p>Instead of trying to prove anything about the action value function Q (and the idea of an action value function <span class="math-container">$Q(s)$</span> which does not evaluate an action), you should start with different fundamental objects, e.g. your (stochastic) policy function is <span class="math-container">$\pi(s'|s): \mathcal{S} \times \mathcal{S} \rightarrow \mathbb{R}$</span>, and the Bellman equation for value function is <span class="math-container">$v_{\pi}(s) = \sum_{s' \in \mathcal{S}} \pi(s'|s)(r(s, s') + \gamma v_{\pi}(s'))$</span></p>
  <p>Note your criteria 2 (that the policy is deterministic) is <em>not</em> required.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-07T17:15:22.883Z
Answers.Text: Neil, was I correct that what OP is suggesting is closely related to QSS learning? I haven't been able to look at the paper I'm thinking of since he has posted, but I really think it is.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-07T17:54:50.060Z
Answers.Text: @David I'm, not sure, but I think it is related, yes. It would be another way to code the action away from the agent's perspective. Using QSS, the agent could see and work with  afterstates, and the inverse dynamics model would convert the agent's state goals into actions. Importantly though, for model-free learning for the agent, the model would also need to provide a list of all possible target states for the next time step (or the agent would need access to an environment model that did this)
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-04-07T13:06:27.377Z
Answers.LastActivityDate: 2024-04-07T14:00:36.233Z
Answers.LastEditDate: 2024-04-07T14:00:36.233Z
Answers.UserDisplayName: Neil Slater
Body: <p>I read <a href="https://ai.stackexchange.com/questions/32464/when-to-use-the-state-value-function-vs-and-when-to-use-the-state-action-val">this post</a>, was thinking about it and now I have a hypothesis but I am not sure whether or not its correct.
  I claim that in Q-learning <span class="math-container">$Q(s,a)$</span> can be replaced by V(s) when</p>
  <ol>
  <li><span class="math-container">$p(s'|a,s)$</span> is deterministic, so <span class="math-container">$p(s'|A=a,S=s) = 1$</span></li>
  <li>the policy is deterministic, so <span class="math-container">$\pi(A \mid S = s)$</span> is just <span class="math-container">$\pi(s)$</span></li>
  <li>the reward function <span class="math-container">$R(s, a, s')$</span> is independent of the action and thus just <span class="math-container">$R(s, s')$</span></li>
  <li>It is possible to enumerate all next states <span class="math-container">$s'$</span> for a given state <span class="math-container">$s$</span>.</li>
  </ol>
  <p>I firstly implemented that as a very basic Deep Q-Learning (or better Deep V-Learning) network and used OpenAI's gym environment <strong>frozen lake.</strong> Frozen lake has deterministic transition probabilities, the states are known and the rewards are bound to the state, independent of the action. So it fulfills my requirements.
  I implemented only one neural network which is used for policy and target prediction. The network outputs only 1 value (V(s) rather than 1 value for each state action pair. I used a replay buffer and employed epsilon greedy. Epsilon greedy yields either a random action or it calculates <span class="math-container">$\max_{s'} V^\pi(s')$</span>.
  It learned an optimal policy so I tried to proof my claim formally.</p>
  <p><strong>Proof:</strong></p>
  <p>Per definition:</p>
  <p><span class="math-container">$Q^\pi(s,a) = \sum_{s'\in S}p(s'|a,s)[R(s, a, s') + \gamma V^\pi(s')]$</span></p>
  <p>Since <span class="math-container">$p(s'|a,s) = 1$</span> and <span class="math-container">$R(s,a,s')=R(s,s')$</span>, action <span class="math-container">$a$</span> vanishes and so:</p>
  <p><span class="math-container">$Q^\pi(s) = \sum_{s'}[R(s,s') + \gamma V^\pi(s')]$</span></p>
  <p>So <span class="math-container">$Q^\pi(s) = Q^\pi(s, a)$</span>.</p>
  <p>Next, per definition:</p>
  <p><span class="math-container">$V^\pi(s) = \sum_{a \in A}\pi(a|s) * Q^\pi(s,a)$</span></p>
  <p>Since <span class="math-container">$\pi(a| s) = 1$</span> we can remove the sum and use <span class="math-container">$Q^\pi(s)$</span>.</p>
  <p>So <span class="math-container">$V^\pi(s) = Q^\pi(s)$</span>.</p>
  <p>Per definition, the Q-formula:</p>
  <p><span class="math-container">$Q^\pi(s,a) \leftarrow Q^\pi(s,a) + \alpha[R(s, a, s') + \gamma \max_a Q^\pi(s',a)- Q^\pi(s,a)]$</span></p>
  <p>Since we showed that <span class="math-container">$Q^\pi(s,a) = V^\pi(s)$</span> and <span class="math-container">$R(s,a,s') = R(s,s')$</span>, we can say:</p>
  <p><span class="math-container">$V^\pi(s) \leftarrow V^\pi(s) + \alpha[R(s,s') + \gamma \max_{s'} V^\pi(s')- V^\pi(s)]$</span></p>
  <p>What do you think?</p>
  <p>UPDATE</p>
  <p><a href="https://gist.github.com/NMO13/92e09fb376d93916f4204542443b47a3" rel="nofollow noreferrer">Here</a> is the code that shows that an optimal policy in Q-learning can be found with just V(s).</p>
  
Tags: <reinforcement-learning><q-learning><value-functions>
Title: Can Q(s,a) be replaced by V(s) when certrain requirements are met?
UserDisplayName: NMO
ThreadText: # Can Q(s,a) be replaced by V(s) when certrain requirements are met?
  
  (Asked by: username_0 on 2024-04-04)
  
  I read [this post](https://ai.stackexchange.com/questions/32464/when-to-use-the-state-value-function-vs-and-when-to-use-the-state-action-val), was thinking about it and now I have a hypothesis but I am not sure whether or not its correct.
  I claim that in Q-learning $Q(s,a)$ can be replaced by V(s) when
  
  1. $p(s'|a,s)$ is deterministic, so $p(s'|A=a,S=s) = 1$
  2. the policy is deterministic, so $\pi(A \mid S = s)$ is just $\pi(s)$
  3. the reward function $R(s, a, s')$ is independent of the action and thus just $R(s, s')$
  4. It is possible to enumerate all next states $s'$ for a given state $s$.
  
  I firstly implemented that as a very basic Deep Q-Learning (or better Deep V-Learning) network and used OpenAI's gym environment **frozen lake.** Frozen lake has deterministic transition probabilities, the states are known and the rewards are bound to the state, independent of the action. So it fulfills my requirements.
  I implemented only one neural network which is used for policy and target prediction. The network outputs only 1 value (V(s) rather than 1 value for each state action pair. I used a replay buffer and employed epsilon greedy. Epsilon greedy yields either a random action or it calculates $\max\_{s'} V^\pi(s')$.
  It learned an optimal policy so I tried to proof my claim formally.
  
  **Proof:**
  
  Per definition:
  
  $Q^\pi(s,a) = \sum\_{s'\in S}p(s'|a,s)[R(s, a, s') + \gamma V^\pi(s')]$
  
  Since $p(s'|a,s) = 1$ and $R(s,a,s')=R(s,s')$, action $a$ vanishes and so:
  
  $Q^\pi(s) = \sum\_{s'}[R(s,s') + \gamma V^\pi(s')]$
  
  So $Q^\pi(s) = Q^\pi(s, a)$.
  
  Next, per definition:
  
  $V^\pi(s) = \sum\_{a \in A}\pi(a|s) \* Q^\pi(s,a)$
  
  Since $\pi(a| s) = 1$ we can remove the sum and use $Q^\pi(s)$.
  
  So $V^\pi(s) = Q^\pi(s)$.
  
  Per definition, the Q-formula:
  
  $Q^\pi(s,a) \leftarrow Q^\pi(s,a) + \alpha[R(s, a, s') + \gamma \max\_a Q^\pi(s',a)- Q^\pi(s,a)]$
  
  Since we showed that $Q^\pi(s,a) = V^\pi(s)$ and $R(s,a,s') = R(s,s')$, we can say:
  
  $V^\pi(s) \leftarrow V^\pi(s) + \alpha[R(s,s') + \gamma \max\_{s'} V^\pi(s')- V^\pi(s)]$
  
  What do you think?
  
  UPDATE
  
  [Here](https://gist.github.com/NMO13/92e09fb376d93916f4204542443b47a3) is the code that shows that an optimal policy in Q-learning can be found with just V(s).
  
  **Tags:** reinforcement-learning, q-learning, value-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-04-07. Score: 1 | Accepted answer
  
  Your criteria #4:
  
  > 4. It is possible to enumerate all next states $s'$ for a given state $s$.
  
  makes me think you are describing *afterstate* representation.
  
  Yes, you can really ignore actions for model-free RL in that case, but it's not a matter of "cancelling out" actions starting from definitions in Q-learning. Instead, actions and action-value functions simply don't exist in this subset of reinforcement learning problems. The agent's "action" is to select the next state directly, there is no separate action space, and nothing need get formally labelled with a separate type of action ($a \in \mathcal{A}$). This is common in board games for example, although these have different complications to deal with adversarial agents.
  
  Instead of trying to prove anything about the action value function Q (and the idea of an action value function $Q(s)$ which does not evaluate an action), you should start with different fundamental objects, e.g. your (stochastic) policy function is $\pi(s'|s): \mathcal{S} \times \mathcal{S} \rightarrow \mathbb{R}$, and the Bellman equation for value function is $v\_{\pi}(s) = \sum\_{s' \in \mathcal{S}} \pi(s'|s)(r(s, s') + \gamma v\_{\pi}(s'))$
  
  Note your criteria 2 (that the policy is deterministic) is *not* required.
  
  > **Comments:**
  > **username_2**: Neil, was I correct that what OP is suggesting is closely related to QSS learning? I haven't been able to look at the paper I'm thinking of since he has posted, but I really think it is.  
  > **username_1**: @David I'm, not sure, but I think it is related, yes. It would be another way to code the action away from the agent's perspective. Using QSS, the agent could see and work with  afterstates, and the inverse dynamics model would convert the agent's state goals into actions. Importantly though, for model-free learning for the agent, the model would also need to provide a list of all possible target states for the next time step (or the agent would need access to an environment model that did this)  
  
  ---
  
  