Answers.Body: <p>A formal POMDP approach that models a belief state (a distribution over possible states) needs to have a theoretical model for the hidden aspects of the state. This is not always appropriate or possible.</p>
  <p>If you have no theoretical MDP that underlies the POMDP (e.g. possible sets of cards an opponent holds in poker), and are limited in not only not knowing hidden data, but also not knowing what structure hidden data might have or the rules governing it, then you need to look at other options.</p>
  <p>One approach is to try to extract maximum information that the full observation history can provide. This being the only data that you have, is the best you can do if you lack a theoretical model. This full history can include all observed state data, actions and rewards.</p>
  <p>If the data can be summarised logically into a fixed size - perhaps with some theoretical justification in the problem domain - you can use that. If not, you can attempt to learn an abstract summary from experience by making your value function, or your policy function, or a separate &quot;helper&quot; function (e.g. that predicts next observables) into a sequence-based predictor such as an LSTM.</p>
  <p>Predicting market behaviour and making investment choices are not known to be easy targets for machine learning models. This is in part because all the low hanging fruit for statistical analysis has been researched and exploited for many years already. There are skilled mathematicians who work in this area professionally, who already understand how to create sophisticated predictive models, and they are your competition for any agent that you could write. It is very hard to create an autonomous agent that can do well in this kind of environment.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T20:07:32.247Z
Answers.Text: Thank you so much for this detailed and insightful response. However, I have a few questions: 1. What do you mean by "summarised logically into a fixed size"? 2. Would you be able to provide a very high level example of the "attempt to learn an abstract summary..." perhaps as applied to this financial example? 3. How do we predict future observables if they depend on our actions taken? do we just add this into our model? Thanks again :)
Answers.UserDisplayName: QMath
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T21:57:38.837Z
Answers.Text: @QMath I think trying to answer those follow-on questions here would make the answer too long. Apart from 3 - yes you should add action history and current action. Action history may easily affect hidden state, and current action very likely to impact next observables. For other follow up questions, they would be good new questions on the site
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-21T22:02:36.317Z
Answers.Text: That's fair regarding the first two questions and great, that makes sense regarding the third. Thanks again!
Answers.UserDisplayName: QMath
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-17T21:36:18.170Z
Answers.LastActivityDate: 2024-02-21T21:59:43.017Z
Answers.LastEditDate: 2024-02-21T21:59:43.017Z
Answers.UserDisplayName: Neil Slater
Body: <p>In the literature that I've seen so far on how to either exactly or approximately solve POMDPs (Partially-Observable Markov Decision Processes), there seems to be a lot of focus placed on maintaining a distribution over all possible states of what the current state is as well as approximating the value function as a function of this distribution at each step.</p>
  <p>Is this level of rigour how things are implemented in practice?</p>
  <p>For example, say we have a market with cash and a single other asset that our agent participates in.</p>
  <p>First, our agent receives an observation:
  <span class="math-container">$o_0 = (t_0, b_0, h_0)$</span> where <span class="math-container">$t_i$</span> denotes the time of the observation, <span class="math-container">$b_i$</span> denotes some actionable observation of the market such as current bid/ask price of the asset, and <span class="math-container">$h_i$</span> denotes our holdings of cash and the other asset.</p>
  <p>Then, our agent chooses an action <span class="math-container">$a_0 \in  \mathcal{A}_0$</span> which is to either sell or buy a specified amount of the asset or do nothing.</p>
  <p>Then our agent receives the next observation <span class="math-container">$o_1 = (t_1, b_1, h_1)$</span>
  and also receives a reward, <span class="math-container">$r_1 = L(h_1, b_1) - L(h_0, b_0)$</span> where <span class="math-container">$L(h_i, b_i)$</span> denotes the value of the portfolio with holdings, <span class="math-container">$h_i$</span>, as determined by the market representation <span class="math-container">$b_i$</span></p>
  <p>This is then repeated for the remainder of the potentially-infinite horizon of the POMDP.</p>
  <p>If our goal is to maximimize the sum of the rewards over this potentially-infinite horizon and we are able to interact with this environment &quot;live&quot; (as in we are not simulating the dynamics, our agent is directly interacting with the market), how would we go about approaching this in practice if it does differ from the theoretical approaches?</p>
  
Tags: <reinforcement-learning><machine-learning><deep-rl><applications><pomdp>
Title: How are POMDPs solved in practice?
UserDisplayName: QMath
ThreadText: # How are POMDPs solved in practice?
  
  (Asked by: username_0 on 2024-02-12)
  
  In the literature that I've seen so far on how to either exactly or approximately solve POMDPs (Partially-Observable Markov Decision Processes), there seems to be a lot of focus placed on maintaining a distribution over all possible states of what the current state is as well as approximating the value function as a function of this distribution at each step.
  
  Is this level of rigour how things are implemented in practice?
  
  For example, say we have a market with cash and a single other asset that our agent participates in.
  
  First, our agent receives an observation:
  $o\_0 = (t\_0, b\_0, h\_0)$ where $t\_i$ denotes the time of the observation, $b\_i$ denotes some actionable observation of the market such as current bid/ask price of the asset, and $h\_i$ denotes our holdings of cash and the other asset.
  
  Then, our agent chooses an action $a\_0 \in \mathcal{A}\_0$ which is to either sell or buy a specified amount of the asset or do nothing.
  
  Then our agent receives the next observation $o\_1 = (t\_1, b\_1, h\_1)$
  and also receives a reward, $r\_1 = L(h\_1, b\_1) - L(h\_0, b\_0)$ where $L(h\_i, b\_i)$ denotes the value of the portfolio with holdings, $h\_i$, as determined by the market representation $b\_i$
  
  This is then repeated for the remainder of the potentially-infinite horizon of the POMDP.
  
  If our goal is to maximimize the sum of the rewards over this potentially-infinite horizon and we are able to interact with this environment "live" (as in we are not simulating the dynamics, our agent is directly interacting with the market), how would we go about approaching this in practice if it does differ from the theoretical approaches?
  
  **Tags:** reinforcement-learning, machine-learning, deep-rl, applications, pomdp
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-17. Score: 2 | Accepted answer
  
  A formal POMDP approach that models a belief state (a distribution over possible states) needs to have a theoretical model for the hidden aspects of the state. This is not always appropriate or possible.
  
  If you have no theoretical MDP that underlies the POMDP (e.g. possible sets of cards an opponent holds in poker), and are limited in not only not knowing hidden data, but also not knowing what structure hidden data might have or the rules governing it, then you need to look at other options.
  
  One approach is to try to extract maximum information that the full observation history can provide. This being the only data that you have, is the best you can do if you lack a theoretical model. This full history can include all observed state data, actions and rewards.
  
  If the data can be summarised logically into a fixed size - perhaps with some theoretical justification in the problem domain - you can use that. If not, you can attempt to learn an abstract summary from experience by making your value function, or your policy function, or a separate "helper" function (e.g. that predicts next observables) into a sequence-based predictor such as an LSTM.
  
  Predicting market behaviour and making investment choices are not known to be easy targets for machine learning models. This is in part because all the low hanging fruit for statistical analysis has been researched and exploited for many years already. There are skilled mathematicians who work in this area professionally, who already understand how to create sophisticated predictive models, and they are your competition for any agent that you could write. It is very hard to create an autonomous agent that can do well in this kind of environment.
  
  > **Comments:**
  > **username_0**: Thank you so much for this detailed and insightful response. However, I have a few questions: 1. What do you mean by "summarised logically into a fixed size"? 2. Would you be able to provide a very high level example of the "attempt to learn an abstract summary..." perhaps as applied to this financial example? 3. How do we predict future observables if they depend on our actions taken? do we just add this into our model? Thanks again :)  
  > **username_1**: @QMath I think trying to answer those follow-on questions here would make the answer too long. Apart from 3 - yes you should add action history and current action. Action history may easily affect hidden state, and current action very likely to impact next observables. For other follow up questions, they would be good new questions on the site  
  > **username_0**: That's fair regarding the first two questions and great, that makes sense regarding the third. Thanks again!  
  
  ---
  
  