Answers.Body: <p>There is no major difference here between a POMDP and MDP. When setting reward values, you are generally trying to give the minimal information to the agent that when the sum of rewards is maximised, it solves the problem that you are posing. </p>
  
  <blockquote>
    <p>In literature it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way you have do define a reward for every possible Action-State combination.</p>
  </blockquote>
  
  <p>Some defined value of reward has to be returned after <em>all</em> state, action pairs taken in the environment. The value could be <span class="math-container">$0$</span> of course. </p>
  
  <p>Reward can depend on, or be a function of, current state, action, next state and a random factor. In a POMDP it may also be from any unobserved factor in the environment (you might know this in a simulation because you have created the environment and are choosing not to share the data with the agent).</p>
  
  <p>In practice, the reward often does not have to depend on all the possible factors. In addition, the relationship between the factors and the possible reward can be very simple or sparse. </p>
  
  <p>Classic examples you may find in the literature include:</p>
  
  <ul>
  <li><p>Reward in a game can be simply <span class="math-container">$+1$</span> for winning, or <span class="math-container">$-1$</span> for losing, granted at the end. All other rewards are <span class="math-container">$0$</span></p></li>
  <li><p>If your goal is to reach a certain state, such as exit from a maze, in minimum time, then a fixed reward of <span class="math-container">$-1$</span> per time step is enough to express the need to minimize the total number of steps.</p></li>
  <li><p>For a goal of maintaining stability and avoiding a failure in e.g. <a href="https://gym.openai.com/envs/CartPole-v0/" rel="nofollow noreferrer">CartPole</a>, then a reward of +1 per time step without failure suffices.</p></li>
  </ul>
  
  <p>All of these examples express the reward function as a simple condition plus one or two numbers. The key thing is that they allow you to <em>calculate</em> a suitable reward on each and every time step. They are all strictly reward functions - the general case can cover very complex functions if you wish, that will depend entirely on the goals you want to set the agent to learn/solve.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-29T20:53:28.597Z
Answers.LastActivityDate: 2018-10-30T16:33:05.633Z
Answers.LastEditDate: 2018-10-30T16:33:05.633Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>In a POMDP, you minimize the following cost function</p>
  <p><span class="math-container">\begin{align}
  J_{\pi_{0:N-1}} (\cdot) 
  &amp;= 
  \mathbb{E} \left( g_N (x_N) + \sum_{k=0}^{N-1} g_k (x_k, \pi_k(\cdot)) \right),
  \end{align}</span>
  where <span class="math-container">$g_N$</span> is the terminal cost and <span class="math-container">$g_k$</span> is the step cost.</p>
  <p>Note that this is only the formal problem definition. If you talk about an actual system, you might have a hard time recognizing that this is connected to your reward. What you observe is formally a value of the function J and practically a simple float</p>
  <p>More details in German are on my blog: <a href="https://martin-thoma.com/probabilistische-planung/#mdp-vs-pomdp-vs-rl_1" rel="nofollow noreferrer">https://martin-thoma.com/probabilistische-planung/#mdp-vs-pomdp-vs-rl_1</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-29T20:01:56.777Z
Answers.LastActivityDate: 2021-01-20T12:45:54.747Z
Answers.LastEditDate: 2021-01-20T12:45:54.747Z
Answers.UserDisplayName: Martin Thoma
Body: <p>How do I define a reward function for my POMDP model?</p>
  
  <p>In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way you have to do define a reward for every possible action-state combination. I think that the examples in the literature might not be practical in reality, but only for the purpose of explanation.</p>
  
Tags: <reinforcement-learning><reward-design><reward-functions><pomdp>
Title: How to define a reward function in POMDPs?
UserDisplayName: Bryan McGill
ThreadText: # How to define a reward function in POMDPs?
  
  (Asked by: username_0 on 2018-10-29)
  
  How do I define a reward function for my POMDP model?
  
  In the literature, it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way you have to do define a reward for every possible action-state combination. I think that the examples in the literature might not be practical in reality, but only for the purpose of explanation.
  
  **Tags:** reinforcement-learning, reward-design, reward-functions, pomdp
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-10-29. Score: 1 | Accepted answer
  
  There is no major difference here between a POMDP and MDP. When setting reward values, you are generally trying to give the minimal information to the agent that when the sum of rewards is maximised, it solves the problem that you are posing.
  
  > In literature it is common to use one simple number as a reward, but I am not sure if this is really how you define a function. Because this way you have do define a reward for every possible Action-State combination.
  
  Some defined value of reward has to be returned after *all* state, action pairs taken in the environment. The value could be $0$ of course.
  
  Reward can depend on, or be a function of, current state, action, next state and a random factor. In a POMDP it may also be from any unobserved factor in the environment (you might know this in a simulation because you have created the environment and are choosing not to share the data with the agent).
  
  In practice, the reward often does not have to depend on all the possible factors. In addition, the relationship between the factors and the possible reward can be very simple or sparse.
  
  Classic examples you may find in the literature include:
  
  * Reward in a game can be simply $+1$ for winning, or $-1$ for losing, granted at the end. All other rewards are $0$
  * If your goal is to reach a certain state, such as exit from a maze, in minimum time, then a fixed reward of $-1$ per time step is enough to express the need to minimize the total number of steps.
  * For a goal of maintaining stability and avoiding a failure in e.g. [CartPole](https://gym.openai.com/envs/CartPole-v0/), then a reward of +1 per time step without failure suffices.
  
  All of these examples express the reward function as a simple condition plus one or two numbers. The key thing is that they allow you to *calculate* a suitable reward on each and every time step. They are all strictly reward functions - the general case can cover very complex functions if you wish, that will depend entirely on the goals you want to set the agent to learn/solve.
  
  
  ---
  
  ### Answer by username_2 on 2018-10-29. Score: 0
  
  In a POMDP, you minimize the following cost function
  
  \begin{align}
  J\_{\pi\_{0:N-1}} (\cdot)
  &=
  \mathbb{E} \left( g\_N (x\_N) + \sum\_{k=0}^{N-1} g\_k (x\_k, \pi\_k(\cdot)) \right),
  \end{align}
  where $g\_N$ is the terminal cost and $g\_k$ is the step cost.
  
  Note that this is only the formal problem definition. If you talk about an actual system, you might have a hard time recognizing that this is connected to your reward. What you observe is formally a value of the function J and practically a simple float
  
  More details in German are on my blog: <https://martin-thoma.com/probabilistische-planung/#mdp-vs-pomdp-vs-rl_1>
  
  
  ---
  
  