Demystifying excessively volatile human learning: A Bayesian persistent prior and a neural approximation
Understanding how humans and animals learn about statistical regularities in stable and volatile environments, and utilize these regularities to make predictions and decisions, is an important problem in neuroscience and psychology. Using a Bayesian modeling framework, specifically the Dynamic Belief Model (DBM), it has previously been shown that humans tend to make the default assumption that environmental statistics undergo abrupt, unsignaled changes, even when environmental statistics are actually stable. Because exact Bayesian inference in this setting, an example of switching state space models, is computationally intensive, a number of approximately Bayesian and heuristic algorithms have been proposed to account for learning/prediction in the brain. Here, we examine a neurally plausible algorithm, a special case of leaky integration dynamics we denote as EXP (for exponential filtering), that is significantly simpler than all previously suggested algorithms except for the delta-learning rule, and which far outperforms the delta rule in approximating Bayesian prediction performance. We derive the theoretical relationship between DBM and EXP, and show that EXP gains computational efficiency by foregoing the representation of inferential uncertainty (as does the delta rule), but that it nevertheless achieves near-Bayesian performance due to its ability to incorporate a "persistent prior" influence unique to DBM and absent from the other algorithms. Furthermore, we show that EXP is comparable to DBM but better than all other models in reproducing human behavior in a visual search task, suggesting that human learning and prediction also incorporates an element of persistent prior. More broadly, our work demonstrates that when observations are information-poor, detecting changes or modulating the learning rate is both difficult and thus unnecessary for making Bayes-optimal predictions.
The authors introduce an approximate model of the Bayesian dynamic belief model (DBM)  that is based on exponential filtering and can produce similar results as the DBM but at lower computational costs and increased neural plausibility. I like the idea of defining more plausible solutions to the puzzle of volatililty in human behavior. Yet this paper somehow left me wanting more.   Here are some suggestions on how to improve it:  - as an author I always dislike reviews that say the present work isn't novel enough myself, but this paper really felt a little like that; I don't think that this is necessarily the authors fault per se and definitely doesn't justify rejecting the paper. However, I think the intro could motivate the models a little more. I think it would make sense to mention some cases where humans are volatile, how DBM captures this behavior and why more plausible implementations based on approximations are useful (for example, mentioning computational and neural constraints).   - some minor comments: sometimes the quotation marks are inverted, sometimes not. some of the equations aren't numbered.  - The experiment is a little opaque to me. I don't really understand what was done there. Is this the data from the Yu & Huang paper? How many participants were there? What was their task and how did they get rewarded? I'm not fully sure what Figure 1c means. I guess it shows that the predictions of DBM and Exp align with the data in terms of which patch is likely on average. Is this the only thing the model can capture here? Are there any inter-individual differences that the model can pick up on? Why is the prediction only about the first fixation? Couldn't this be captured already by accounting for some level of stickiness based on what was previously right? How does this relate to volatility of participants' belief if it will be the same option again? I think this part needs more explanation  - In general, I was really excited about the paper when I saw the title, but then it is never really explained how this actually demystifies volatile learning. Doesn't the DBM model already demystify volatility and the current model is a more neurally plausible approximation to that model? My concern about this paper is that it might be a little too niche for the general NIPS community as it is about a more plausible approximation to a model that has been proposed before, so people would essentially have to know the prior model, too. I don't think that's a bad idea and incremental work should be encouraged of course, but then I would really try and emphasize how this goes beyond DBM, perhaps even thinking about a novel prediction it could generate.  - equivalent to a particular form of leaky-integrating neuronal dynamics [19]. -> I think the general NIPS audience would need another sentence here to explain why this is important.  - I wonder how exponential filtering relates to other filtering approaches such as Kalman filtering or Harrison Ritz' control theoretic model of learning in changing envrionments.