Answers.Body: <p>Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in some cases change the priority of solutions.</p>
  
  <p>As a simple example, consider a choice between trajectories with costs A(0,4,4,4) and B(1,1,1,1). In the original cost formula B is clearly better, with 4 total cost compared with A's cost of 12 - A just has one low cost at the beginning, which I put in deliberately as it exposes the problem with your conversion.</p>
  
  <p>In your two reward formulae:</p>
  
  <pre><code>reward = 1/(cost+1)**2. 
    A: 1.0 + 0.04 + 0.04 + 0.04 = 1.12
    B: 0.25 + 0.25 + 0.25 + 0.25 = 1.0
  
  reward = 1/(cost+1).
    A: 1.0 + 0.2 + 0.2 + 0.2 = 1.6
    B: 0.5 + 0.5 + 0.5 + 0.5 = 2.0
  </code></pre>
  
  <p>So with this example (numbers carefully chosen), maximising the total reward favours A for sum of inverse squares but B for sum of inverses, whilst B should be the clear preference for minimising sum of costs. It is possible to find examples for both of your formulae where the best sum of rewards does not give you the lowest cost.</p>
  
  <p>In your case, if you truly want to minimise total cost, then your conversion to rewards should be:</p>
  
  <pre><code>reward = -cost
  </code></pre>
  
  <p>Anything else is technically changing the nature of the problem, and will result in different solutions that may not be optimal with respect to your initial goal.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-15T10:14:55.917Z
Answers.Text: Thank you for that illustration. The reward function you have proposed works fairly well with a neural  actor-critic, but when am using a Q-table, it returns a poorer convergence (with higher  global cost) than my two functions. What do you think is the matter? I initialized Q-values with zeros.
Answers.UserDisplayName: EArwa
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-15T11:24:09.883Z
Answers.Text: @EArwa: Sorry I don't know. I suggest you ask a new question about that if you are stuck.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-14T09:21:46.807Z
Answers.LastActivityDate: 2020-05-14T09:48:11.583Z
Answers.LastEditDate: 2020-05-14T09:48:11.583Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but there is some difference I get in the global cost convergence for different orders of the reward function. If I use the reward function as:</p>
  <p><span class="math-container">$$\text{reward} = \frac{1}{(\text{cost}+1)^2}$$</span></p>
  <p>the algorithm converges better (lower global cost, which is the objective of the process) than when I use the reward as:</p>
  <p><span class="math-container">$$\text{reward} = \frac{1}{(\text{cost}+1)}$$</span></p>
  <p>What could be the explanation for this difference? Is it the issue of optimism in the face of uncertainty?</p>
  
Tags: <reinforcement-learning><q-learning><rewards><reward-design><reward-functions>
Title: Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?
UserDisplayName: EArwa
ThreadText: # Why is the reward function $\text{reward} = 1/{(\text{cost}+1)^2}$ better than $\text{reward} =1/(\text{cost}+1)$?
  
  (Asked by: username_0 on 2020-05-14)
  
  I have implemented a simple Q-learning algorithm to minimize a cost function by setting the reward to the inverse of the cost of the action taken by the agent. The algorithm converges nicely, but there is some difference I get in the global cost convergence for different orders of the reward function. If I use the reward function as:
  
  $$\text{reward} = \frac{1}{(\text{cost}+1)^2}$$
  
  the algorithm converges better (lower global cost, which is the objective of the process) than when I use the reward as:
  
  $$\text{reward} = \frac{1}{(\text{cost}+1)}$$
  
  What could be the explanation for this difference? Is it the issue of optimism in the face of uncertainty?
  
  **Tags:** reinforcement-learning, q-learning, rewards, reward-design, reward-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-14. Score: 4 | Accepted answer
  
  Reinforcement learning (RL) control maximises the expected sum of rewards. If you change the reward metric, it will change what counts as optimal. Your reward functions are not the same, so will in some cases change the priority of solutions.
  As a simple example, consider a choice between trajectories with costs A(0,4,4,4) and B(1,1,1,1). In the original cost formula B is clearly better, with 4 total cost compared with A's cost of 12 - A just has one low cost at the beginning, which I put in deliberately as it exposes the problem with your conversion.
  In your two reward formulae:
  reward = 1/(cost+1)**2. 
    A: 1.0 + 0.04 + 0.04 + 0.04 = 1.12
    B: 0.25 + 0.25 + 0.25 + 0.25 = 1.0
  
  reward = 1/(cost+1).
    A: 1.0 + 0.2 + 0.2 + 0.2 = 1.6
    B: 0.5 + 0.5 + 0.5 + 0.5 = 2.0
  
  So with this example (numbers carefully chosen), maximising the total reward favours A for sum of inverse squares but B for sum of inverses, whilst B should be the clear preference for minimising sum of costs. It is possible to find examples for both of your formulae where the best sum of rewards does not give you the lowest cost.
  In your case, if you truly want to minimise total cost, then your conversion to rewards should be:
  reward = -cost
  
  Anything else is technically changing the nature of the problem, and will result in different solutions that may not be optimal with respect to your initial goal.
  
  > **Comments:**
  > **username_0**: Thank you for that illustration. The reward function you have proposed works fairly well with a neural  actor-critic, but when am using a Q-table, it returns a poorer convergence (with higher  global cost) than my two functions. What do you think is the matter? I initialized Q-values with zeros.  
  > **username_1**: @EArwa: Sorry I don't know. I suggest you ask a new question about that if you are stuck.  
  
  ---
  
  