Answers.Body: <p>Your overall reasoning sounds reasonable. In a known finite episodic task, while the choice of <span class="math-container">$\gamma$</span> could be seen as a global hyperparameter as you narrated, the impact of <span class="math-container">$\gamma$</span> is more localized within each episode and its choice is to influence the agent's exploration-exploitation tradeoff based on the characteristics of its specific task and the environment. There is no one-size-fits-all value for <span class="math-container">$\gamma$</span>. For instance, a short-term swing trader's <span class="math-container">$\gamma$</span> must be smaller than a long term investor's, on the other hand a hyperparameter of a model is generally systematically tuned by some grid search like method to achieve possibly best training and test performance without such prior insight.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-05T04:55:20.390Z
Answers.LastActivityDate: 2024-03-05T04:55:20.390Z
Answers.UserDisplayName: cinch
Body: <p>Sutton and Barto explains that in an continuing task, <span class="math-container">$\gamma$</span> is used to make the reward not infinite.  <strong>In a known finite episodic task, does <span class="math-container">$\gamma$</span> retain any value?</strong>  It is said that changing <span class="math-container">$\gamma$</span> changes how near-sighted an agent is but that seems more like hyperparameterization to suit a given task, not a specific strength.</p>
  <p>For example:</p>
  <p>If we consider a multi-arm bandit with two states, a <span class="math-container">$\gamma$</span> of zero is fine because one only needs to maximize the present decision.  If the agent was only given 10000 pulls, a <span class="math-container">$\gamma$</span> of 1 might be noisier but the actions in a state should approach their actual values on a long enough timeline.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-05T13:52:58.347Z
Comments.Text: Yes. The use of discount factor helps reduce the variance when computing the multi-step return. Future rewards which are more uncertain receive a higher discount.
Comments.UserDisplayName: pi-tau
Tags: <reinforcement-learning><discount-factor>
Title: In a known finite episodic task, is there still value to discount?
UserDisplayName: foreverska
ThreadText: # In a known finite episodic task, is there still value to discount?
  
  (Asked by: username_0 on 2024-03-05)
  
  Sutton and Barto explains that in an continuing task, $\gamma$ is used to make the reward not infinite. **In a known finite episodic task, does $\gamma$ retain any value?** It is said that changing $\gamma$ changes how near-sighted an agent is but that seems more like hyperparameterization to suit a given task, not a specific strength.
  
  For example:
  
  If we consider a multi-arm bandit with two states, a $\gamma$ of zero is fine because one only needs to maximize the present decision. If the agent was only given 10000 pulls, a $\gamma$ of 1 might be noisier but the actions in a state should approach their actual values on a long enough timeline.
  
  **Tags:** reinforcement-learning, discount-factor
  
  > **Comments:**
  > **username_1**: Yes. The use of discount factor helps reduce the variance when computing the multi-step return. Future rewards which are more uncertain receive a higher discount.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-03-05. Score: 1 | Accepted answer
  
  Your overall reasoning sounds reasonable. In a known finite episodic task, while the choice of $\gamma$ could be seen as a global hyperparameter as you narrated, the impact of $\gamma$ is more localized within each episode and its choice is to influence the agent's exploration-exploitation tradeoff based on the characteristics of its specific task and the environment. There is no one-size-fits-all value for $\gamma$. For instance, a short-term swing trader's $\gamma$ must be smaller than a long term investor's, on the other hand a hyperparameter of a model is generally systematically tuned by some grid search like method to achieve possibly best training and test performance without such prior insight.
  
  
  ---
  
  