Answers.Body: <p>RL is currently being applied to environments which are definitely not markovian, maybe they are weakly markovian with decreasing dependency.</p>
  <p>You need to provide details of your problem, if it is 1 step then any optimization system can be used.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-26T05:31:45.963Z
Answers.Text: what do you mean by 1 step? Currently yes, the simulator can take a set of input and render output. But there is nothing dynamic and time-dependent in it. Like I mentioned the next state (output) then is only being determined here by the next action (input) which I am free to choose and as such has got nothing to do with the previous state.
Answers.UserDisplayName: chupa_kabra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-26T05:23:21.667Z
Answers.LastActivityDate: 2021-03-26T05:23:21.667Z
Answers.UserDisplayName: FourierFlux
Answers.Body: <p>I view it as a generalization of the conditional Markovian case.
  It does have the Markov property, in that the future state depends solely on the input at the given state, which probably is to be sampled from a stochastic policy, that is conditioned on the current state.</p>
  <p>It seems to me to be a more general, simpler, and unconstrained case.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-12T10:50:47.953Z
Answers.LastActivityDate: 2022-03-12T10:50:47.953Z
Answers.UserDisplayName: Jakob G
Body: <p>I will start working on a project where we want to optimize the production of a chemical unit through reinforcement learning approach. From the SME's, we already obtained a simulator code that can take some input and render us the output. A part of our output is our objective function that we want to maximize by tuning the input variables. From a reinforcement learning angle, the inputs will be the agent actions, while the state and reward can be obtained from the output. We are currently in the process of building a RL environment, the major part of which is the simulator code described above.</p>
  <p>We were talking to a RL expert and she mentioned that one of the thing that we have here conceptually wrong is that our environment will not have the Markov property in the sense that it is really a 'one-step process' with the process not continuing from the previous state and there is no sort of continuity in state transitions. She is correct there. This made me think, how can we get around this then. Can we perhaps append some part of the current state to the next state etc. More importantly, I have seen RL applied to optimal control in other examples as well which are non-markovian ex. scheduling, tsp problems, process optimization etc. What is the explanation in such cases? Does one simply assumes process to be markovian with unknown transition function?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-26T18:57:05.780Z
Comments.Text: The Markov Property is an extremely hard constraint that is often not met in real life application. Many RL methods will work fine without the Markov property. A simple way is to add previous states in the value and action fonctions for instance.
  Given that, I think that you should either give more details on your specific problem or rephrase the theoretical part you are not understanding.
Comments.UserDisplayName: Romain
Tags: <reinforcement-learning><q-learning><deep-rl><markov-decision-process><markov-property>
Title: Reinforcement Learning for an environment that is non-markovian
UserDisplayName: chupa_kabra
ThreadText: # Reinforcement Learning for an environment that is non-markovian
  
  (Asked by: username_0 on 2021-03-26)
  
  I will start working on a project where we want to optimize the production of a chemical unit through reinforcement learning approach. From the SME's, we already obtained a simulator code that can take some input and render us the output. A part of our output is our objective function that we want to maximize by tuning the input variables. From a reinforcement learning angle, the inputs will be the agent actions, while the state and reward can be obtained from the output. We are currently in the process of building a RL environment, the major part of which is the simulator code described above.
  
  We were talking to a RL expert and she mentioned that one of the thing that we have here conceptually wrong is that our environment will not have the Markov property in the sense that it is really a 'one-step process' with the process not continuing from the previous state and there is no sort of continuity in state transitions. She is correct there. This made me think, how can we get around this then. Can we perhaps append some part of the current state to the next state etc. More importantly, I have seen RL applied to optimal control in other examples as well which are non-markovian ex. scheduling, tsp problems, process optimization etc. What is the explanation in such cases? Does one simply assumes process to be markovian with unknown transition function?
  
  **Tags:** reinforcement-learning, q-learning, deep-rl, markov-decision-process, markov-property
  > **Comments:**
  > **username_1**: The Markov Property is an extremely hard constraint that is often not met in real life application. Many RL methods will work fine without the Markov property. A simple way is to add previous states in the value and action fonctions for instance. Given that, I think that you should either give more details on your specific problem or rephrase the theoretical part you are not understanding.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-03-26. Score: 2 | Accepted answer
  
  RL is currently being applied to environments which are definitely not markovian, maybe they are weakly markovian with decreasing dependency.
  
  You need to provide details of your problem, if it is 1 step then any optimization system can be used.
  
  > **Comments:**
  > **username_0**: what do you mean by 1 step? Currently yes, the simulator can take a set of input and render output. But there is nothing dynamic and time-dependent in it. Like I mentioned the next state (output) then is only being determined here by the next action (input) which I am free to choose and as such has got nothing to do with the previous state.  
  
  ---
  
  ### Answer by username_3 on 2022-03-12. Score: 0
  
  I view it as a generalization of the conditional Markovian case.
  It does have the Markov property, in that the future state depends solely on the input at the given state, which probably is to be sampled from a stochastic policy, that is conditioned on the current state.
  
  It seems to me to be a more general, simpler, and unconstrained case.
  
  
  ---
  
  