Answers.Body: <p>My understanding from your question is that you have the following data generated from a random policy:</p>
  <p><span class="math-container">$$[s_0, s_1, s_2 . . . s_n]$$</span></p>
  <p>That is, the state observed at each time step.</p>
  <p>You know nothing more about the MDP, such as the transition or reward functions. Although the MDP is discrete and fully observable (and thus usual RL theory is supported), you do not have any observations other than a single list of states.</p>
  <p>You wish to obtain an estimate for <span class="math-container">$Q(s_j, a_j)$</span>, given a specific state/action pair that has not been observed before. This is not possible. You do not have any data from your observations regarding rewards, nor the actions taken by the random agent. At minimum, you would need data about rewards and actions that were observed. Observed rewards are needed to calculate returns, and value functions are expected returns. Observed actions are needed to assign returns to the correct <span class="math-container">$(s,a)$</span> pair when calculating estimates.</p>
  <p>In addition, if you want to make a meaningful estimate for an as-yet unseen state/action pair, then you would need to be using some form of function approximation for the action value function Q. Otherwise your estimate will be whatever default estimate you had assigned at the beginning of learning.</p>
  <p>What inferences <em>can</em> you make given the data you have? Very roughly:</p>
  <p><span class="math-container">$$p(s_{k+1} | s_k, \pi ) \gt 0, \forall k \in [0,n-1]$$</span></p>
  <p>i.e. the state sequence that was observed gives you proof of non-zero transition probabilities between those states under the given policy. This is not enough to learn a value function, but might have other uses.</p>
  <p>With repeated observations - multiple lists of observed states - you could infer what those probabilities were, but would not be able to separate the effect of action choice from state transition rules (because the action was not observed). If you added reward observations then you could learn a state value function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-03T23:46:12.010Z
Answers.LastActivityDate: 2021-04-05T13:35:27.343Z
Answers.LastEditDate: 2021-04-05T13:35:27.343Z
Answers.UserDisplayName: Neil Slater
Body: <p>Assume you are given a sequence of states followed by the agent, generated by a random policy, <span class="math-container">$[s_0, s_1, s_2, \dots,  s_n]$</span>. Furthermore, assume the MDP is fully observable and time is discrete.</p>
  <p>Is it possible to find the Q-value for a state-action pair <span class="math-container">$(s_j, a_j)$</span> which was not encountered along this sequence?</p>
  <p>From my understanding of the MDP, yes, it would be possible. However, I'm unsure how to get this Q-value.</p>
  
Tags: <reinforcement-learning><q-learning><markov-decision-process><value-functions>
Title: Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence?
UserDisplayName: Sbean
ThreadText: # Given a sequence of states followed by the agent, is it possible to find the Q-value for a state-action pair not in this sequence?
  
  (Asked by: username_0 on 2021-04-03)
  
  Assume you are given a sequence of states followed by the agent, generated by a random policy, $[s\_0, s\_1, s\_2, \dots, s\_n]$. Furthermore, assume the MDP is fully observable and time is discrete.
  
  Is it possible to find the Q-value for a state-action pair $(s\_j, a\_j)$ which was not encountered along this sequence?
  
  From my understanding of the MDP, yes, it would be possible. However, I'm unsure how to get this Q-value.
  
  **Tags:** reinforcement-learning, q-learning, markov-decision-process, value-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-03. Score: 1
  
  My understanding from your question is that you have the following data generated from a random policy:
  
  $$[s\_0, s\_1, s\_2 . . . s\_n]$$
  
  That is, the state observed at each time step.
  
  You know nothing more about the MDP, such as the transition or reward functions. Although the MDP is discrete and fully observable (and thus usual RL theory is supported), you do not have any observations other than a single list of states.
  
  You wish to obtain an estimate for $Q(s\_j, a\_j)$, given a specific state/action pair that has not been observed before. This is not possible. You do not have any data from your observations regarding rewards, nor the actions taken by the random agent. At minimum, you would need data about rewards and actions that were observed. Observed rewards are needed to calculate returns, and value functions are expected returns. Observed actions are needed to assign returns to the correct $(s,a)$ pair when calculating estimates.
  
  In addition, if you want to make a meaningful estimate for an as-yet unseen state/action pair, then you would need to be using some form of function approximation for the action value function Q. Otherwise your estimate will be whatever default estimate you had assigned at the beginning of learning.
  
  What inferences *can* you make given the data you have? Very roughly:
  
  $$p(s\_{k+1} | s\_k, \pi ) \gt 0, \forall k \in [0,n-1]$$
  
  i.e. the state sequence that was observed gives you proof of non-zero transition probabilities between those states under the given policy. This is not enough to learn a value function, but might have other uses.
  
  With repeated observations - multiple lists of observed states - you could infer what those probabilities were, but would not be able to separate the effect of action choice from state transition rules (because the action was not observed). If you added reward observations then you could learn a state value function.
  
  
  ---
  
  