Answers.Body: <blockquote>
  <p>How would you implement this &quot;Number of Steps&quot; cost?</p>
  </blockquote>
  <p>What the paper is referring to is the reward <em>discounting</em> process which is a standard way of formulating RL problems, either continuous ones, or episodic ones where the goal is to complete a task in the least time (in the episodic version, a fixed cost per time step will also achieve this).</p>
  <p>As such, this usually is implemented in the formulation of the value function calculations. The discount factor is usually represented as gamma, <span class="math-container">$\gamma$</span>.</p>
  <p>For Q-learning, the factor should be in the TD target calculation:</p>
  <p><span class="math-container">$$G_{t:t+1} = R_{t+1} + \gamma \max_{a'}Q(S_{t+1},a')$$</span></p>
  <p>For Monte Carlo control, the factor appears more like this in the calculation of a return:</p>
  <p><span class="math-container">$$G_t = \sum_{k=0}^{T-t} \gamma^k R_{t+k+1}$$</span></p>
  <blockquote>
  <p>would it be best to use an exponential functions to discount the reward at the current time step?</p>
  </blockquote>
  <p>Essentially that is what normal discounting is - an exponential decay of future reward. But if you have implemented &quot;normal&quot; Q-learning from equations like those above, it should already be there.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-07T06:41:02.177Z
Answers.LastActivityDate: 2020-09-28T06:32:15.500Z
Answers.LastEditDate: 2020-09-28T06:32:15.500Z
Answers.UserDisplayName: Neil Slater
Body: <p>I am currently implementing the paper <a href="https://web.archive.org/web/20180205094740/http://web.engr.illinois.edu/%7Eslazebni/publications/iccv15_active.pdf" rel="nofollow noreferrer">Active Object Localization with Deep Reinforcement Learning</a> in Python. While reading about the reward scheme I came across the following:</p>
  <blockquote>
  <p>Finally, the proposed reward scheme implicitly considers the number of steps as a cost because of the way in which Q-learning models the discount of future rewards (positive and negative).</p>
  </blockquote>
  <p>How would you implement this &quot;number of steps&quot; cost? I am keeping track of the number of steps that have been taken, therefore would it be best to use an exponential functions to discount the reward at the current time step?</p>
  <p>If anyone has a good idea or knows the standard in regard to this I would love to hear your thoughts.</p>
  
Tags: <reinforcement-learning><ai-design><papers><rewards>
Title: How should I take into consideration the number of steps in the reward function?
UserDisplayName: Wizard
ThreadText: # How should I take into consideration the number of steps in the reward function?
  
  (Asked by: username_0 on 2018-07-07)
  
  I am currently implementing the paper [Active Object Localization with Deep Reinforcement Learning](https://web.archive.org/web/20180205094740/http://web.engr.illinois.edu/%7Eslazebni/publications/iccv15_active.pdf) in Python. While reading about the reward scheme I came across the following:
  
  > Finally, the proposed reward scheme implicitly considers the number of steps as a cost because of the way in which Q-learning models the discount of future rewards (positive and negative).
  
  How would you implement this "number of steps" cost? I am keeping track of the number of steps that have been taken, therefore would it be best to use an exponential functions to discount the reward at the current time step?
  
  If anyone has a good idea or knows the standard in regard to this I would love to hear your thoughts.
  
  **Tags:** reinforcement-learning, ai-design, papers, rewards
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-07-07. Score: 3 | Accepted answer
  
  > How would you implement this "Number of Steps" cost?
  
  What the paper is referring to is the reward *discounting* process which is a standard way of formulating RL problems, either continuous ones, or episodic ones where the goal is to complete a task in the least time (in the episodic version, a fixed cost per time step will also achieve this).
  
  As such, this usually is implemented in the formulation of the value function calculations. The discount factor is usually represented as gamma, $\gamma$.
  
  For Q-learning, the factor should be in the TD target calculation:
  
  $$G\_{t:t+1} = R\_{t+1} + \gamma \max\_{a'}Q(S\_{t+1},a')$$
  
  For Monte Carlo control, the factor appears more like this in the calculation of a return:
  
  $$G\_t = \sum\_{k=0}^{T-t} \gamma^k R\_{t+k+1}$$
  
  > would it be best to use an exponential functions to discount the reward at the current time step?
  
  Essentially that is what normal discounting is - an exponential decay of future reward. But if you have implemented "normal" Q-learning from equations like those above, it should already be there.
  
  
  ---
  
  