Answers.Body: <p>To my knowledge, there does not exist anything along the lines of model-based reinforcement learning with time-sensitive data. I think the best chance you have is to try to abstract the data that you have into a model which is not time-sensitive.</p>
  <p>What would happen when you get past the timestamps of your original data? I am guessing that when testing this model, you will have states with time stamps past your original latest timestamp. Or are you only using your model in the time stamps of your training data? Is it going to be a useful model then?</p>
  <p>Firstly, please note that the <a href="https://ai.stackexchange.com/help/on-topic">topics</a> of this forum are mostly theoretical. This question leans on the boundary. If you frame your question a bit differently, ending your post with a clear question and also putting that question in the title, you will get more responses.</p>
  <p>I feel like the question is 'What are the possibilities for model-based reinforcement learning for time-sensitive data?' But this question is inherently vague as it implies that you have a model according to the definitions of models in RL, which are not time sensitive. If you would like a better answer then please provide more information or ask your question on a more practical forum if it is more practical than what I guessed.</p>
  <p>Disclaimer: would have just put some comments under the post as this is not a real sufficient answer, but my reputation is not high enough for that :P</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-28T09:58:36.727Z
Answers.LastActivityDate: 2020-12-28T09:58:36.727Z
Answers.UserDisplayName: Robin van Hoorn
Answers.Body: <p>It seems that your description perfectly matches the naive RL approach. What you can do with model-based RL is perform rollouts with the model to predict future states. In other words, with an accurate model you might predict the next state <em><strong>X(t + 1)</strong></em> given the current state <em><strong>X(t)</strong></em> and applying some action <em><strong>a(t)</strong></em>. The following (and recently) <a href="https://arxiv.org/pdf/2104.06294.pdf" rel="nofollow noreferrer">paper</a> shows how these rollouts works.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-24T08:47:23.523Z
Answers.LastActivityDate: 2022-01-24T08:47:23.523Z
Answers.UserDisplayName: HenDoNR
Body: <p>I have time-series data. When I take an action, it impacts the next state, because my action directly determines the next state, but it is not known what the impact is.</p>
  <p>To be concrete: I have <span class="math-container">$X(t)$</span> and <span class="math-container">$a(t-1)$</span>, where <span class="math-container">$X(t)$</span> is n-dimensional time-series data and <span class="math-container">$a(t)$</span> is 1-dimensional time-series data. At time <span class="math-container">$t$</span>, they together represent the observation/state space. Also, at time <span class="math-container">$t$</span>, the agent makes a decision about <span class="math-container">$a(t)$</span>. This decision (action) <span class="math-container">$a(t)$</span> directly defines the next state space <span class="math-container">$X(t+1)$</span> and rewards, by some function <span class="math-container">$f$</span> &amp; <span class="math-container">$g$</span>, <span class="math-container">$f(a(t), X(t)) = X(t+1)$</span> and <span class="math-container">$g(a(t), X(t)) = R(t+1)$</span>.</p>
  <p>I have to estimate this impact, i.e. where I will end up (what will be the next state). I decided to use a model-based RL algorithm, because, from my knowledge, model-based RL does exactly this.</p>
  <p>Can you advise me on a good paper and Github code, to implement this project?</p>
  <p>As I noticed, there do not exist many works on Model-based RL.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-30T05:17:41.587Z
Comments.Text: DeepMind's [MuZero](https://arxiv.org/abs/1911.08265) algorithm learns both a model and policy using RL.
Comments.UserDisplayName: DavidJ
Tags: <reinforcement-learning><reference-request><time-series><model-based-methods>
Title: Model-based RL for time series data
UserDisplayName: Mariam 
ThreadText: # Model-based RL for time series data
  
  (Asked by: username_0 on 2020-12-28)
  
  I have time-series data. When I take an action, it impacts the next state, because my action directly determines the next state, but it is not known what the impact is.
  
  To be concrete: I have $X(t)$ and $a(t-1)$, where $X(t)$ is n-dimensional time-series data and $a(t)$ is 1-dimensional time-series data. At time $t$, they together represent the observation/state space. Also, at time $t$, the agent makes a decision about $a(t)$. This decision (action) $a(t)$ directly defines the next state space $X(t+1)$ and rewards, by some function $f$ & $g$, $f(a(t), X(t)) = X(t+1)$ and $g(a(t), X(t)) = R(t+1)$.
  
  I have to estimate this impact, i.e. where I will end up (what will be the next state). I decided to use a model-based RL algorithm, because, from my knowledge, model-based RL does exactly this.
  
  Can you advise me on a good paper and Github code, to implement this project?
  
  As I noticed, there do not exist many works on Model-based RL.
  
  **Tags:** reinforcement-learning, reference-request, time-series, model-based-methods
  > **Comments:**
  > **username_1**: DeepMind's [MuZero](https://arxiv.org/abs/1911.08265) algorithm learns both a model and policy using RL.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-12-28. Score: 0
  
  To my knowledge, there does not exist anything along the lines of model-based reinforcement learning with time-sensitive data. I think the best chance you have is to try to abstract the data that you have into a model which is not time-sensitive.
  
  What would happen when you get past the timestamps of your original data? I am guessing that when testing this model, you will have states with time stamps past your original latest timestamp. Or are you only using your model in the time stamps of your training data? Is it going to be a useful model then?
  
  Firstly, please note that the [topics](https://ai.stackexchange.com/help/on-topic) of this forum are mostly theoretical. This question leans on the boundary. If you frame your question a bit differently, ending your post with a clear question and also putting that question in the title, you will get more responses.
  
  I feel like the question is 'What are the possibilities for model-based reinforcement learning for time-sensitive data?' But this question is inherently vague as it implies that you have a model according to the definitions of models in RL, which are not time sensitive. If you would like a better answer then please provide more information or ask your question on a more practical forum if it is more practical than what I guessed.
  
  Disclaimer: would have just put some comments under the post as this is not a real sufficient answer, but my reputation is not high enough for that :P
  
  
  ---
  
  ### Answer by username_3 on 2022-01-24. Score: 0
  
  It seems that your description perfectly matches the naive RL approach. What you can do with model-based RL is perform rollouts with the model to predict future states. In other words, with an accurate model you might predict the next state ***X(t + 1)*** given the current state ***X(t)*** and applying some action ***a(t)***. The following (and recently) [paper](https://arxiv.org/pdf/2104.06294.pdf) shows how these rollouts works.
  
  
  ---
  
  