Answers.Body: <p>It is possible, at design time for a reinforcement learning problem, to allow for changes within an environment. You can make any element into a variable property of the state, that the agent can realistically be told at the start or sense from the environment.</p>
  <p>If you do add new variable to model the possibility of change:</p>
  <ul>
  <li><p>It allows the agent to learn to solve a more general problem where the chosen property can vary.</p>
  </li>
  <li><p>It increases the size of the state space.</p>
  </li>
  <li><p>It requires training to include variations of the new variable.</p>
  </li>
  </ul>
  <p>Usually this also increases the time taken to train.</p>
  <p>It is not always possible to use a state variable for the task - perhaps a goal state is effectively hidden from the agent and the purpose of training is for it to be discovered. In which case, you will require at least some re-training. It may be faster to start with the existing trained agent if the difference is not large.</p>
  <p>If you cannot simply extend the state representation, and the environment changes in a small enough way, then it may also be possible to use an agent which continuously explores which will re-train itself over time in repsonse to changes in the environment. The <a href="https://zhangruochi.com/Dyna-Q-and-Dyna-Q/2020/09/30/" rel="nofollow noreferrer">DynaQ+ algorithm</a> is an example of a method which is designed to explore and find changes in the agent's environment to allow for this kind of online retraining when things change.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-03T08:53:47.383Z
Answers.LastActivityDate: 2021-05-03T08:53:47.383Z
Answers.UserDisplayName: Neil Slater
Body: <p>I was reading up a paper that did routing based on an MDP, and I was wondering because, in routing, there is a sender node and a receiver node, so if the receiver node changes (sending a message to someone else), would we have to train the MDP algorithm all over again?</p>
  <p>This also got me thinking about what would happen even if one node in the process of transmission changes. Does using an MDP for training the agent mean that the obstacle and goals should never change?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-03T08:29:20.447Z
Comments.Text: Technically you do not "train" an MDP, unless you are asking about inverse reinforcement learning. I do not think you are, but not 100% sure, so please clarify. Usually you train an **agent** which learns to optimise reward when making action choices within an MDP. In which case your question is about whether to retrain the **agent** as opposed to the MDP
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-03T08:31:57.583Z
Comments.Text: Made the relevant changes, thanks for pointing it out
Comments.UserDisplayName: Ravish Jha
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-03T12:17:23.270Z
Comments.Text: To have more context, could you please provide the name of and link to the paper you were reading? Moreover, you may want to reformulate your question because saying "an agent trained using MDP" or "using an MDP for training" does not make much sense, i.e., even after your last changes, the post is still not "ok", in my view. The MDP is just the mathematical model of the environment. The solution to the MDP is a policy. RL algorithms are used to find a policy. So, based on this info, you may want to formulate your post again to make it more meaningful.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><deep-rl><markov-decision-process>
Title: What would happen to an agent trained using Markov Decision Process if the goal node changes?
UserDisplayName: Ravish Jha
ThreadText: # What would happen to an agent trained using Markov Decision Process if the goal node changes?
  
  (Asked by: username_0 on 2021-05-03)
  
  I was reading up a paper that did routing based on an MDP, and I was wondering because, in routing, there is a sender node and a receiver node, so if the receiver node changes (sending a message to someone else), would we have to train the MDP algorithm all over again?
  
  This also got me thinking about what would happen even if one node in the process of transmission changes. Does using an MDP for training the agent mean that the obstacle and goals should never change?
  
  **Tags:** reinforcement-learning, deep-rl, markov-decision-process
  > **Comments:**
  > **username_1**: Technically you do not "train" an MDP, unless you are asking about inverse reinforcement learning. I do not think you are, but not 100% sure, so please clarify. Usually you train an **agent** which learns to optimise reward when making action choices within an MDP. In which case your question is about whether to retrain the **agent** as opposed to the MDP  
  > **username_0**: Made the relevant changes, thanks for pointing it out  
  > **username_2**: To have more context, could you please provide the name of and link to the paper you were reading? Moreover, you may want to reformulate your question because saying "an agent trained using MDP" or "using an MDP for training" does not make much sense, i.e., even after your last changes, the post is still not "ok", in my view. The MDP is just the mathematical model of the environment. The solution to the MDP is a policy. RL algorithms are used to find a policy. So, based on this info, you may want to formulate your post again to make it more meaningful.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-05-03. Score: 2 | Accepted answer
  
  It is possible, at design time for a reinforcement learning problem, to allow for changes within an environment. You can make any element into a variable property of the state, that the agent can realistically be told at the start or sense from the environment.
  
  If you do add new variable to model the possibility of change:
  
  * It allows the agent to learn to solve a more general problem where the chosen property can vary.
  * It increases the size of the state space.
  * It requires training to include variations of the new variable.
  
  Usually this also increases the time taken to train.
  
  It is not always possible to use a state variable for the task - perhaps a goal state is effectively hidden from the agent and the purpose of training is for it to be discovered. In which case, you will require at least some re-training. It may be faster to start with the existing trained agent if the difference is not large.
  
  If you cannot simply extend the state representation, and the environment changes in a small enough way, then it may also be possible to use an agent which continuously explores which will re-train itself over time in repsonse to changes in the environment. The [DynaQ+ algorithm](https://zhangruochi.com/Dyna-Q-and-Dyna-Q/2020/09/30/) is an example of a method which is designed to explore and find changes in the agent's environment to allow for this kind of online retraining when things change.
  
  
  ---
  
  