Log-normality and Skewness of Estimated State/Action Values in Reinforcement Learning
Under/overestimation of state/action values are harmful for reinforcement learning agents. In this paper, we show that a state/action value estimated using the Bellman equation can be decomposed to a weighted sum of path-wise values that follow log-normal distributions. Since log-normal distributions are skewed, the distribution of estimated state/action values can also be skewed, leading to an imbalanced likelihood of under/overestimation. The degree of such imbalance can vary greatly among actions and policies within a single problem instance, making the agent prone to select actions/policies that have inferior expected return and higher likelihood of overestimation. We present a comprehensive analysis to such skewness, examine its factors and impacts through both theoretical and empirical results, and discuss the possible ways to reduce its undesirable effects.
This paper focuses on the problem arising from skewness in the distribution of value estimates, which may result in over- or under-estimation. With careful analysis, the paper shows that a particular model-based value estimate is approximately log-normally distributed, which is skewed and thus leading to the possibility of over- or under-estimation. It is further shown that positive and negative rewards induce opposite sort of skewness. With simple experiments, the problem of over/underestimation is illustrated.
This is an interesting paper with some interesting insights on over/underestimation of values. I would like the reviewers to clarify the following two issues.
The results are implied for a particular kind of model-based value estimation method, more specifically, one which relies on definition 3.1. The implication of this result to other kinds of estimation methods, especially the model-free ones is not clear. A discussion is required on why a similar effect is expected with other kinds of values estimation method.
It is disappointing to have no experiments with the suggested remedies, such as the balancing the impact of positive and negative rewards and reward shaping. Some experiments are expected here to see whether these are indeed helpful. And an elaborate discussion of reward balancing and how to achieve it could also improve the quality of the paper.