Body: <p>Suppose <span class="math-container">$x_{t+1} \sim \mathbb{P}(\cdot | x_t, a_t)$</span> denotes the state transition dynamics in a reinforcement learning (RL) problem. Let <span class="math-container">$y_{t+1} = \mathbb{P}(\cdot | x_{t+1})$</span> denote the noisy observation or the imperfect state information. Let <span class="math-container">$H_{t}$</span> denote the history of actions and observations <span class="math-container">$H_{t+1} = \{b_0,y_0,a_0,\cdots,y_{t+1}\}$</span>.</p>
  
  <p>For the RL Partially Observed Markov Decision Process (RL-POMDP), the summary of the history is contained in the "belief state"  <span class="math-container">$b_{t+1}(i) = \mathbb{P}(x_{t+1} = i | H_{t+1})$</span>, which is the posterior distribution over the states conditioned on the history.</p>
  
  <p>Now, suppose the model is NOT known. Clearly, the belief state can't be computed.</p>
  
  <p>Can we use a Gaussian Process (GP) to approximate the belief distribution <span class="math-container">$b_{t}$</span> at every instant <span class="math-container">$t$</span>?</p>
  
  <p>Can Variational GP be adapted to such a situation? Can universal approximation property of GP be invoked here?</p>
  
  <p>Are there such results in the literature? </p>
  
  <p>Any references and insights into this problem would be much appreciated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-12T09:58:16.373Z
Comments.Text: How? If you have no model you literally know nothing. You have no idea how to assign a probability to any state or even what a state is.
Comments.UserDisplayName: FourierFlux
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-12T15:09:32.977Z
Comments.Text: @FourierFlux: Say you have access to the following (training) data: Input {a_t,y_t} and output {x_t}. It's just during the implementation (testing), we only have access to imperfect information. Could a GP, which captures the distribution over the states, match the belief state? at every time t? Are there such works?
Comments.UserDisplayName: math_phile
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-13T10:25:32.693Z
Comments.Text: @math_phile You need a model for that. GP is a model. You said you have no model. You need to make some assumptions about what you want to predict.
Comments.UserDisplayName: Tomas
Tags: <reinforcement-learning><markov-decision-process><pomdp><bayesian-optimization><gaussian-process>
Title: Can we use a Gaussian process to approximate the belief distribution at every instant in a POMDP?
UserDisplayName: math_phile
ThreadText: # Can we use a Gaussian process to approximate the belief distribution at every instant in a POMDP?
  
  (Asked by: username_0 on 2020-05-12)
  
  Suppose $x\_{t+1} \sim \mathbb{P}(\cdot | x\_t, a\_t)$ denotes the state transition dynamics in a reinforcement learning (RL) problem. Let $y\_{t+1} = \mathbb{P}(\cdot | x\_{t+1})$ denote the noisy observation or the imperfect state information. Let $H\_{t}$ denote the history of actions and observations $H\_{t+1} = \{b\_0,y\_0,a\_0,\cdots,y\_{t+1}\}$.
  
  For the RL Partially Observed Markov Decision Process (RL-POMDP), the summary of the history is contained in the "belief state" $b\_{t+1}(i) = \mathbb{P}(x\_{t+1} = i | H\_{t+1})$, which is the posterior distribution over the states conditioned on the history.
  
  Now, suppose the model is NOT known. Clearly, the belief state can't be computed.
  
  Can we use a Gaussian Process (GP) to approximate the belief distribution $b\_{t}$ at every instant $t$?
  
  Can Variational GP be adapted to such a situation? Can universal approximation property of GP be invoked here?
  
  Are there such results in the literature?
  
  Any references and insights into this problem would be much appreciated.
  
  **Tags:** reinforcement-learning, markov-decision-process, pomdp, bayesian-optimization, gaussian-process
  > **Comments:**
  > **username_1**: How? If you have no model you literally know nothing. You have no idea how to assign a probability to any state or even what a state is.  
  > **username_0**: @FourierFlux: Say you have access to the following (training) data: Input {a_t,y_t} and output {x_t}. It's just during the implementation (testing), we only have access to imperfect information. Could a GP, which captures the distribution over the states, match the belief state? at every time t? Are there such works?  
  > **username_2**: @math_phile You need a model for that. GP is a model. You said you have no model. You need to make some assumptions about what you want to predict.  
  
  ---
  
  ## 0 answers
  
  