Answers.Body: <p>The problem is the definition of what's <span class="math-container">$o_t$</span> and <span class="math-container">$s_t$</span>:</p>
  <ul>
  <li><span class="math-container">$o_t$</span> is the (partially) observable part</li>
  <li><span class="math-container">$s_t$</span> is ideally a perfect model of the world</li>
  </ul>
  <p>now, by definition, you don't have access directly to <span class="math-container">$s_t$</span>, otherwise half of the problem would be solved on the fly</p>
  <p>So, answering your questions:</p>
  <ol>
  <li>Because there is no way to learn it, instead as proposed in the video, you use a NN to approximate a posterior, which takes as input all the previous observations and actions, and gives you a useful state representation (think about an RNN as encoder, the final latent vector is the state representation)</li>
  <li>because you want to learn a state representation that carries information about your future, which is <span class="math-container">$o_t$</span> (think about the RNN encoder, I give you the first N frames of a video, you give me the N+1 frame, so the last &quot;latent vector&quot; <span class="math-container">$s_t$</span> should carry information useful to predict what's about to happen <span class="math-container">$o_t$</span>)</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-14T17:24:11.283Z
Answers.Text: @platoDev no prob, good luck with the course
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-15T04:09:13.177Z
Answers.Text: Thanks to your answer, it now seems obvious to me that observation is the only thing that is observable thus a reconstruction term should appear in the objective function. However, I have come to another question. What would be the possible interpretation of maximizing Σ(_|_)∗(_t|_t) where summation is over s_t? I still cannot get a grasp of it. It certainly is not likelihood.
Answers.UserDisplayName: platoDev
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-15T10:54:14.433Z
Answers.Text: @platoDev can you please post a new question with more informations and reference from where you see this notation? thanks
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-14T11:09:30.477Z
Answers.LastActivityDate: 2023-09-14T11:09:30.477Z
Answers.UserDisplayName: Alberto
Body: <p>I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley.</p>
  <p>At <a href="https://youtu.be/2EWicx9uP1Q?si=wnzkUnXf3CteGcdy" rel="nofollow noreferrer">1:52</a> of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model <span class="math-container">$p(o_t | s_t)$</span> instead of a state model <span class="math-container">$p(s_t | o_t)$</span>? For me, it seems more intuitive to get a state from a state model <span class="math-container">$p(s_t|o_t)$</span> and use a state transition model to plan. Is there a specific reason to learn an observation model instead of a state model?</p>
  <p>Also at <a href="https://youtu.be/2EWicx9uP1Q?si=rLubm356Vi9q5ScV" rel="nofollow noreferrer">4:38</a> of the lecture, following objective function is presented.</p>
  <p><span class="math-container">$max_\phi \frac{1}{N} \Sigma_{i=1}^{N} \Sigma_{t=1}^{T} E_{(s_t, s_{t+1}) \sim p(s_t, s_{t+1}|o_{1:T}, a_{1:T})}[logp_\phi(s_{t+1,i} | s_{t,i}, a_{t, i}) + logp_\phi(o_{t, i} | s_{t, i})]$</span></p>
  <p>In the objective function, <span class="math-container">$logp_\phi(o_t | s_t)$</span> is included. I understand that maximizing this term could be desirable since it can be interpreted as minimizing a reconstruction loss. However, I cannot clearly see why <span class="math-container">$p_\phi(o_t|s_t)$</span> would be preferable over <span class="math-container">$p_\phi(s_t|o_t)$</span> as an objective function term.</p>
  <p>I tried to interpret <span class="math-container">$p_\phi(o_t|s_t)$</span> term in the objective function as a maximum log likelihood and thus justify having it in the loss instead of <span class="math-container">$p_\phi(s_t|o_t)$</span>. However, I do not think that it can be interpreted as a log likelihood since <span class="math-container">$E_{s_t \sim p_\phi(s_t|o_t)}[logp_\phi(o_{t, i} | s_{t, i})] = \Sigma_{s_{t,i}} logp_\phi(o_{t, i} | s_{t, i}) * p_\phi(s_{t,i}|o_{t,i}) \neq logp_\phi(o_{t,i})$</span></p>
  <p>In summary, my questions are following.</p>
  <ol>
  <li>Why learn an observation model <span class="math-container">$p_\phi(o_t|s_t)$</span> instead of a state model <span class="math-container">$p_\phi(s_t|o_t)$</span></li>
  <li>Why have the reconstruction term in the loss instead of <span class="math-container">$logp_\phi(s_t|o_t)$</span>?</li>
  </ol>
  <p>Thank you</p>
  
Tags: <reinforcement-learning><deep-rl><objective-functions><model-based-methods>
Title: why learn an observation model when training latent space model in model based rl
UserDisplayName: platoDev
ThreadText: # why learn an observation model when training latent space model in model based rl
  
  (Asked by: username_0 on 2023-09-14)
  
  I'm currently studying reinforcement learning through CS 285 provided by UC Berkeley.
  
  At [1:52](https://youtu.be/2EWicx9uP1Q?si=wnzkUnXf3CteGcdy) of the part 5 of the lecture 11, I got confused on why one would want to learn an observation model $p(o\_t | s\_t)$ instead of a state model $p(s\_t | o\_t)$? For me, it seems more intuitive to get a state from a state model $p(s\_t|o\_t)$ and use a state transition model to plan. Is there a specific reason to learn an observation model instead of a state model?
  
  Also at [4:38](https://youtu.be/2EWicx9uP1Q?si=rLubm356Vi9q5ScV) of the lecture, following objective function is presented.
  
  $max\_\phi \frac{1}{N} \Sigma\_{i=1}^{N} \Sigma\_{t=1}^{T} E\_{(s\_t, s\_{t+1}) \sim p(s\_t, s\_{t+1}|o\_{1:T}, a\_{1:T})}[logp\_\phi(s\_{t+1,i} | s\_{t,i}, a\_{t, i}) + logp\_\phi(o\_{t, i} | s\_{t, i})]$
  
  In the objective function, $logp\_\phi(o\_t | s\_t)$ is included. I understand that maximizing this term could be desirable since it can be interpreted as minimizing a reconstruction loss. However, I cannot clearly see why $p\_\phi(o\_t|s\_t)$ would be preferable over $p\_\phi(s\_t|o\_t)$ as an objective function term.
  
  I tried to interpret $p\_\phi(o\_t|s\_t)$ term in the objective function as a maximum log likelihood and thus justify having it in the loss instead of $p\_\phi(s\_t|o\_t)$. However, I do not think that it can be interpreted as a log likelihood since $E\_{s\_t \sim p\_\phi(s\_t|o\_t)}[logp\_\phi(o\_{t, i} | s\_{t, i})] = \Sigma\_{s\_{t,i}} logp\_\phi(o\_{t, i} | s\_{t, i}) \* p\_\phi(s\_{t,i}|o\_{t,i}) \neq logp\_\phi(o\_{t,i})$
  
  In summary, my questions are following.
  
  1. Why learn an observation model $p\_\phi(o\_t|s\_t)$ instead of a state model $p\_\phi(s\_t|o\_t)$
  2. Why have the reconstruction term in the loss instead of $logp\_\phi(s\_t|o\_t)$?
  
  Thank you
  
  **Tags:** reinforcement-learning, deep-rl, objective-functions, model-based-methods
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-14. Score: 2 | Accepted answer
  
  The problem is the definition of what's $o\_t$ and $s\_t$:
  
  * $o\_t$ is the (partially) observable part
  * $s\_t$ is ideally a perfect model of the world
  
  now, by definition, you don't have access directly to $s\_t$, otherwise half of the problem would be solved on the fly
  
  So, answering your questions:
  
  1. Because there is no way to learn it, instead as proposed in the video, you use a NN to approximate a posterior, which takes as input all the previous observations and actions, and gives you a useful state representation (think about an RNN as encoder, the final latent vector is the state representation)
  2. because you want to learn a state representation that carries information about your future, which is $o\_t$ (think about the RNN encoder, I give you the first N frames of a video, you give me the N+1 frame, so the last "latent vector" $s\_t$ should carry information useful to predict what's about to happen $o\_t$)
  
  > **Comments:**
  > **username_1**: @username_0 no prob, good luck with the course  
  > **username_0**: Thanks to your answer, it now seems obvious to me that observation is the only thing that is observable thus a reconstruction term should appear in the objective function. However, I have come to another question. What would be the possible interpretation of maximizing Σ(_|_)∗(_t|_t) where summation is over s_t? I still cannot get a grasp of it. It certainly is not likelihood.  
  > **username_1**: @username_0 can you please post a new question with more informations and reference from where you see this notation? thanks  
  
  ---
  
  