Why so gloomy? A Bayesian explanation of human pessimism bias in the multi-armed bandit task
How humans make repeated choices among options with imperfectly known reward outcomes is an important problem in psychology and neuroscience. This is often studied using multi-armed bandits, which is also frequently studied in machine learning. We present data from a human stationary bandit experiment, in which we vary the average abundance and variability of reward availability (mean and variance of the reward rate distribution). Surprisingly, we find subjects significantly underestimate prior mean of reward rates -based on their self-report on their reward expectation of non-chosen arms at the end of a game. Previously, human learning in the bandit task was found to be well captured by a Bayesian ideal learning model, the Dynamic Belief Model (DBM), albeit under an incorrect generative assumption of the temporal structure -humans assume reward rates can change over time even though they are truly fixed. We find that the "pessimism bias" in the bandit task is well captured by the prior mean of DBM when fitted to human choices; but it is poorly captured by the prior mean of the Fixed Belief Model (FBM), an alternative Bayesian model that (correctly) assumes reward rates to be constants. This pessimism bias is also incompletely captured by a simple reinforcement learning model (RL) commonly used in neuroscience and psychology, in terms of fitted initial Q-values. While it seems sub-optimal, and thus mysterious, that humans have an underestimated prior reward expectation, our simulations show that an underestimated prior mean helps to maximize long-term gain, if the observer assumes volatility when reward rates are stable, and utilizes a softmax decision policy instead of the optimal one (obtainable by dynamic programming). This raises the intriguing possibility that the brain underestimates reward rates to compensate for the incorrect non-stationarity assumption in the generative model and a simplified decision policy.
I have updated my score in light of the proposed revisions. I like this paper and would be happy if this gets accepted.  I liked this paper. The authors show that participants underestimate prior reward rates in a bandit task, a behavior which is well-captured by their dynamic belief model, but not by a fixed belief model and only partially by a traditional RL-model. Interestingly, this initial pessimism might actually be adpative under the assumption that participants expect non-stationary environments and map expectatios onto choices by using a softmax-policy.  - I like the experimental paradigm. I would have maybe shown the fishing reports as an icon array with green and red dots just to make it comparable to the actual task. Was there any spatial correlation between participants' mean estimates for different options? It could be that they thought closer options on the lake might produce similar outcomes.  - this paradigm also reminds me of a paradigm recently applied by Gershman, in a paper called "Uncertainty and exploration", in which he marked different options as high or low variance to assess participants' exploration strategies  - I wonder how this relates to other models of human exploration, for example the finding that humans are often well-captchured by a combination between a Kalman filter learning algorithm and a Thompson sampling exploration strategy (for example, Speekenbrink & Konstantinidis, 2015). Thompson sampling is also known to be equivalent to probability matching, a heuristic that has been frequently called suboptimal but can actually perform quite well under the assumption of nonstationary environment. This also relates back to converging evidence that participants might use a combination of direct and random exploration in multi-armed bandit tasks, and I'm not sure how that can be incoporated by the DBM. I guess a direct incorporation of information gain in the model is similar to a directed exploration strategy, whereas the softmax parameter might track (value-based) random exploration.   - would have been nice to ask participants abut their confidence in the estimated reward rate too, just to see if this is more based on uncertainty of the environment or more deliberate  - how were participants rewarded in this task?   - as a side note, Wu et al. (Generalization and exploration in vast spaces) found that participants tend to under-estimate the spatial correlation of rewards and that this "pessimism bias" turned out to be beneficial, even for algorithms of Bayesian optimization; I think this is quite similar to your temporal finding  Minor comments:  - I find the coloring scheme for some of the plots a little confusing, sometimes purple is FBM, sometimes it's human data; I would try and make this consistent across plots - I'm not sure how the error bars were calculated, is this a mean over participant-nested estimates?  - line 79: p < 10 − 3 should be $10^{-3}$ - Figure 3: I would make all of the plots equi-sized; I'm also not sure how the models were run here, just with the overall mean or with a mean per participant? I guess it's the overall mean, but I would perhaps say that - References: make sure that journal names are capitalized appropriately, for example "computational biology" should be "Computational Biology"