Reinforcement learning (RL) typically defines a discount factor (γ) as part of the Markov Decision Process. The discount factor values future rewards by an exponential scheme that leads to theoretical convergence guarantees of the Bellman equation. However, evidence from psychology, economics and neuroscience suggests that humans and animals instead have hyperbolic time-preferences ( 1 1+kt for k > 0). Here we extend earlier work of Kurth-Nelson and Redish and propose an efficient deep reinforcement learning agent that acts via hyperbolic discounting and other non-exponential discount mechanisms. We demonstrate that a simple approach approximates hyperbolic discount functions while still using familiar temporal-difference learning techniques in RL. Additionally, and independent of hyperbolic discounting, we make a surprising discovery that simultaneously learning value functions over multiple time-horizons is an effective auxiliary task which often improves over state-of-the-art methods.
The paper investigates hyperbolic discounting as a more biologically plausible alternative to exponential discounting in reinforcement learning. First, it formulates a notion of hazard in MDPs as constant exponential discounting and shows that hyperbolic discounting is consistent with uncertainty over the hazard rate. The paper then shows how value functions learned with exponential discounting can be used to approximate value functions with other forms of discounting. Specifically, the paper shows in section 4 how exponentially-discounted value functions can be used to approximate hyperbolically discounted value functions. The paper then presents experiments on a small MDP and Atari 2600 games, showing that learning discounted action values with many different discount rates as an auxiliary task improves performance on most Atari games.
Overall, I very slightly tend towards accepting this paper for publication. While the idea of hyperbolic discounting didn't seem to pay off in terms of performance on the Atari 2600 games, the idea still has some merit as being more biologically plausible than exponential discounting, and may perform better on a more suitable environment. In addition, the discovery that learning many action-value functions with different exponential discounting rates as auxiliary tasks improves performance is quite interesting.
Notes:
- The footnote on the first page seems to just trail off. It would be better to explicitly state what the takeaway is. I'm guessing humans and animals in general would prefer \$1M now in the first scenario, but \$1.1M in the second scenario?
- The idea of 1/(1-gamma) being an "effective horizon" seems questionable. For gamma=0.9 the "effective horizon" (aka expected number of timesteps before termination) would be 1/(1-.9)=10 timesteps. However, 0.9^(10)=~0.3486 means ~34% of the probability density is still to the right of that number, so calling 1/(1-gamma) an "effective horizon" seems to violate what we normally mean by "horizon".
- A single hazard rate for the entire environment seems counterintuitive. Some states in life are definitely more hazardous than others, which suggests state-dependent discounting might be an interesting idea to explore.
- Many people in the field think we shouldn't even be doing discounting at all (section 10.4 of Reinforcement Learning by Sutton and Barto), only total reward for episodic problems and average reward for continuing problems.
- Pathworld experiments seem like they don't show anything useful; the hyperbolic discounting matches the true hazard behaviour of the environment (i.e., the hazard rate is drawn from a distribution) so it performs better than exponential discounting. However, if the environment had a fixed hazard rate then exponential discounting would perform better.
- Using only 3 random seeds does not seem like enough for the experiments. Also there is no measure of standard error or statistical significance on the graphs.
- Introducing a hyperparameter that increases computation and memory does not really seem like it's solving the problem, and certainly not "efficiently" as claimed in the abstract.
- How was the value of the new hyperparameter set for the experiments? Were several values tried? More importantly, how can a good value for the new hyperparameter be chosen without prior knowledge of the environment?
- Atari 2600 games don't really seem like a great environment to showcase the benefits of hyperbolic discounting.
- It would be interesting to see a comparison with existing auxiliary task approaches, but due to space constraints should probably be left for future work.