General Discounting versus Average Reward
<|reference_start|>General Discounting versus Average Reward: Consider an agent interacting with an environment in cycles. In every interaction cycle the agent is rewarded for its performance. We compare the average reward U from cycle 1 to m (average value) with the future discounted reward V from cycle k to infinity (discounted value). We consider essentially arbitrary (non-geometric) discount sequences and arbitrary reward sequences (non-MDP environments). We show that asymptotically U for m->infinity and V for k->infinity are equal, provided both limits exist. Further, if the effective horizon grows linearly with k or faster, then existence of the limit of U implies that the limit of V exists. Conversely, if the effective horizon grows linearly with k or slower, then existence of the limit of V implies that the limit of U exists.<|reference_end|>
@article{hutter2006general,
    title={General Discounting versus Average Reward},
    author={Marcus Hutter},
    journal={Proc. 17th International Conf. on Algorithmic Learning Theory (ALT
  2006) pages 244-258},
    year={2006},
    number={IDSIA-11-06},
    archivePrefix={arXiv},
    eprint={cs/0605040},
    primaryClass={cs.LG}
}