Answers.Body: <p>The state of an RL agent, in the naive case, just has to be a comprehensive description of the system, or at least the most comprehensive available (in some cases, it's impossible to have a complete description)</p>
  <p>For this reason, you can give any information to the agent, even some random information that is completely irrelevant (though it will probably hinder the performance)</p>
  <p>In very simple terms, consider a person doing that job and ask yourself which information that person needs to make an informed decision on the task you want to solve: that's the set of information you want to give to your agent.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-12T10:33:15.380Z
Answers.LastActivityDate: 2024-08-12T10:33:15.380Z
Answers.UserDisplayName: Alberto
Body: <p>I'm working on a simulation model using RL to optimize an objective function. I'm trying to understand if I need to select my state variables such that I can write state update equations for each one using the state variables that are provided to the RL agent.</p>
  <p>For example, I have a variable for &quot;time to arrival&quot; (<span class="math-container">$T_{arrival}$</span>) that's calculated based on the vehicle's current location and destination under the hood (in the simulation). If I don’t include location and destination in the state space, can I still use <span class="math-container">$T_{arrival}$</span> in my RL model, even though in this case I can't write a state update equation for it directly?</p>
  <p>Similarly, for a state variable like &quot;earliest time a charging station is available&quot; (<span class="math-container">$T_{charge}^{avail}$</span>), do I need to include charge start and end times for each vehicle in the state space to be able to update/calculate <span class="math-container">$T_{charge}^{avail}$</span>? Or can the simulation environment calculate <span class="math-container">$T_{charge}^{avail}$</span> and provide it to the RL agent without violating the principles of dynamic programming?</p>
  <p>Ultimately, do I need to ensure that every state variable in my RL model can be updated directly using other state variables in the state space? or the simulation calculated state variables are enough?</p>
  
Tags: <reinforcement-learning><deep-rl><markov-decision-process><pomdp><hidden-markov-model>
Title: Do State Variables in RL Models Need Direct Update Equations?
UserDisplayName: RookieScientist
ThreadText: # Do State Variables in RL Models Need Direct Update Equations?
  
  (Asked by: username_0 on 2024-08-11)
  
  I'm working on a simulation model using RL to optimize an objective function. I'm trying to understand if I need to select my state variables such that I can write state update equations for each one using the state variables that are provided to the RL agent.
  
  For example, I have a variable for "time to arrival" ($T\_{arrival}$) that's calculated based on the vehicle's current location and destination under the hood (in the simulation). If I don’t include location and destination in the state space, can I still use $T\_{arrival}$ in my RL model, even though in this case I can't write a state update equation for it directly?
  
  Similarly, for a state variable like "earliest time a charging station is available" ($T\_{charge}^{avail}$), do I need to include charge start and end times for each vehicle in the state space to be able to update/calculate $T\_{charge}^{avail}$? Or can the simulation environment calculate $T\_{charge}^{avail}$ and provide it to the RL agent without violating the principles of dynamic programming?
  
  Ultimately, do I need to ensure that every state variable in my RL model can be updated directly using other state variables in the state space? or the simulation calculated state variables are enough?
  
  **Tags:** reinforcement-learning, deep-rl, markov-decision-process, pomdp, hidden-markov-model
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-08-12. Score: 1
  
  The state of an RL agent, in the naive case, just has to be a comprehensive description of the system, or at least the most comprehensive available (in some cases, it's impossible to have a complete description)
  
  For this reason, you can give any information to the agent, even some random information that is completely irrelevant (though it will probably hinder the performance)
  
  In very simple terms, consider a person doing that job and ask yourself which information that person needs to make an informed decision on the task you want to solve: that's the set of information you want to give to your agent.
  
  
  ---
  
  