Answers.Body: <p>You can use function approximation like neural networks to learn the whole environment, i.e. both the transition function, <span class="math-container">$p(s'\mid s, a)$</span>, and the reward model, <span class="math-container">$r(s,a,s')$</span>:
  <span class="math-container">$$p(s',r\mid s,a)$$</span>
  In principle, you can learn it just by plain supervised learning in which given a state-action pair <span class="math-container">$(s,a)$</span> the prediction targets are the next state, <span class="math-container">$s'$</span>, and reward, <span class="math-container">$r$</span>. Once you learn a suitable model you can use it to replace your actual environment, or even to <em>plan</em> (i.e., query the learned simulator instead of the true env to see what would happen next.), and then use the usual model-free RL to learn a policy.</p>
  <p>In your case I think that the most effective model-based algorithms are the ones based on either <a href="http://arxiv.org/abs/1803.10122" rel="nofollow noreferrer">world models</a> or <a href="https://arxiv.org/pdf/1912.01603.pdf" rel="nofollow noreferrer">dreamer</a>. These approaches learn a generative model (e.g., a VAE) to mimic the response of the true env, and after, or at same time, also learn a control policy.</p>
  <p>The thing you want to keep in mind is that the learned simulator is always flawed, meaning that at each timestep it will make a (small) error that cumulate with time. So there are other approaches in which the simulator is used for <span class="math-container">$K$</span> steps at most, or that consider the uncertainty in the predictions such that to account for the mis-modelling of the learned model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-25T21:11:24.787Z
Answers.LastActivityDate: 2023-07-25T21:11:24.787Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am interested in learning how transition probabilities/mdps are constructed in continuous state and action space model-based learning setting. There is some literature available on this matter, but they do not explicitly construct the model to simulate the environment, through policy gradient.</p>
  <p>The closest literature that builds a model that I found is on continuous state space and finite action space, which is slightly different than what I aim to do. Furthermore even in this case the main problem that I face is, transition probability is assumed to give you the probability of obtaning next state, which does not necessarily have to be non-zero for instance when the transition probability is non-atomic transition probability.</p>
  <p>I will appreciate if someone atleast points me relevant literature in this direction.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-25T09:24:39.057Z
Comments.Text: I don't see any question here. Can you please ask a **specific question**?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-25T11:22:29.223Z
Comments.Text: _but they do not explicitly construct the model to simulate the environment_ probably because it's not that trivial... you can approximate it with variational models, but still you will have a very hard time building a model that works in the continuous space (model in the RL setting, thus a env simulator)
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-25T11:24:15.240Z
Comments.Text: but maybe something like this might give a glance https://arxiv.org/abs/1509.02971 (however, not theoretical definition of a continuous MDP is given)
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-25T15:22:45.100Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Tags: <reinforcement-learning><model-based-methods><continuous-action-spaces><continuous-state-spaces>
Title: Model-based learning in continuous state and action spaces
UserDisplayName: hogger
ThreadText: # Model-based learning in continuous state and action spaces
  
  (Asked by: username_0 on 2023-07-24)
  
  I am interested in learning how transition probabilities/mdps are constructed in continuous state and action space model-based learning setting. There is some literature available on this matter, but they do not explicitly construct the model to simulate the environment, through policy gradient.
  
  The closest literature that builds a model that I found is on continuous state space and finite action space, which is slightly different than what I aim to do. Furthermore even in this case the main problem that I face is, transition probability is assumed to give you the probability of obtaning next state, which does not necessarily have to be non-zero for instance when the transition probability is non-atomic transition probability.
  
  I will appreciate if someone atleast points me relevant literature in this direction.
  
  **Tags:** reinforcement-learning, model-based-methods, continuous-action-spaces, continuous-state-spaces
  
  > **Comments:**
  > **username_1**: I don't see any question here. Can you please ask a **specific question**?  
  > **username_2**: _but they do not explicitly construct the model to simulate the environment_ probably because it's not that trivial... you can approximate it with variational models, but still you will have a very hard time building a model that works in the continuous space (model in the RL setting, thus a env simulator)  
  > **username_2**: but maybe something like this might give a glance https://arxiv.org/abs/1509.02971 (however, not theoretical definition of a continuous MDP is given)  
  > **username_3**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2023-07-25. Score: 2
  
  You can use function approximation like neural networks to learn the whole environment, i.e. both the transition function, $p(s'\mid s, a)$, and the reward model, $r(s,a,s')$:
  $$p(s',r\mid s,a)$$
  In principle, you can learn it just by plain supervised learning in which given a state-action pair $(s,a)$ the prediction targets are the next state, $s'$, and reward, $r$. Once you learn a suitable model you can use it to replace your actual environment, or even to *plan* (i.e., query the learned simulator instead of the true env to see what would happen next.), and then use the usual model-free RL to learn a policy.
  
  In your case I think that the most effective model-based algorithms are the ones based on either [world models](http://arxiv.org/abs/1803.10122) or [dreamer](https://arxiv.org/pdf/1912.01603.pdf). These approaches learn a generative model (e.g., a VAE) to mimic the response of the true env, and after, or at same time, also learn a control policy.
  
  The thing you want to keep in mind is that the learned simulator is always flawed, meaning that at each timestep it will make a (small) error that cumulate with time. So there are other approaches in which the simulator is used for $K$ steps at most, or that consider the uncertainty in the predictions such that to account for the mis-modelling of the learned model.
  
  
  ---
  
  