Forgetful Bayes and myopic planning: Human learning and decision-making in a bandit setting
How humans achieve long-term goals in an uncertain environment, via repeated trials and noisy observations, is an important problem in cognitive science. We investigate this behavior in the context of a multi-armed bandit task. We compare human behavior to a variety of models that vary in their representational and computational complexity. Our result shows that subjects' choices, on a trial-totrial basis, are best captured by a "forgetful" Bayesian iterative learning model [21] in combination with a partially myopic decision policy known as Knowledge Gradient [7]. This model accounts for subjects' trial-by-trial choice better than a number of other previously proposed models, including optimal Bayesian learning and risk minimization, ε-greedy and win-stay-lose-shift. It has the added benefit of being closest in performance to the optimal Bayesian model than all the other heuristic models that have the same computational complexity (all are significantly less complex than the optimal model). These results constitute an advancement in the theoretical understanding of how humans negotiate the tension between exploration and exploitation in a noisy, imperfectly known environment.
This paper seeks to understand human behavior in 
      multi-armed bandit problems by comparing human performance to the 
      predictions of four existing models. After presenting a comparison of the 
      models’ predictions to human judgments, the authors conclude that the 
      “knowledge gradient” (Gupta & Miescke, 1996; Frazier, Powell, & 
      Dayanik, 2008) explains human behavior well if it assumes that reward 
      distributions may change over time. The paper also includes an analysis of 
      some trial-by-trial trends in the data and model predictions. 
      --Quality-- I didn't find any errors or conceptual 
      problems in the model evaluation or the experimental design. 
      --Clarity-- The writing is quite clear in general, with 
      some exceptions: * On line 71 “Conditioned on the previous…” this 
      sentence might lead a reader to think authors were claiming that KG is 
      optimal in general. * Section 3.3’s introduction made be think that 
      the evaluation would be based directly on likelihoods of the human 
      judgments under each model, and I had to re-read it after looking at the 
      results. --Originality-- To my knowledge, this is the 
      first paper to compare multiple models against human behavior in 
      multi-armed bandit problems, with the aim understanding the trade-off 
      between exploration and exploitation. That said, the experimental design 
      and the models -- aside from the DBM/FBM comparison -- are not new, and 
      past bandit experiments have considered how humans approach the 
      explore/exploit trade-off (e.g., Ejova, Navarro & Perfors, 2009). 
      --Significance-- My main worry about this paper is that it 
      won't attract a wide audience at NIPS, because the conclusions aren't very 
      far-reaching and it doesn't appear to have content that would draw an 
      algorithms- or applications-oriented audience. Ultimately, the conclusion 
      is that a particular model, KG, explains human behavior better than some 
      plausible alternatives, and does better when it incorporates the 
      assumption that bandits' rewards might change over time. It doesn't make a 
      compelling case that the dynamic belief model is necessary in general, and 
      there's past work that could be taken to suggest that people approach 
      bandit problems as if expecting a changing world (e.g., Shin & Ariely, 
      2004). In a longer version of this paper, I'd be excited to see 
      (1) a new model that addresses the mismatch between KG and human 
      behavior; (2) a larger-scale evaluation using both 0/1 and continuous 
      losses, multiple horizons, and whether or not the rewards are actually 
      static; (3) a look at transfer learning and how performance changes 
      across problem presentations. --Other notes-- There 
      were a few things that I wish had been included in the paper or done a bit 
      differently. * It would have been helpful to see a visualization of 
      how well the models fit different individuals’ judgments, e.g., with error 
      histograms. * I wouldn't have used an ANOVA's failure-to-reject to 
      claim there's no difference between parameter estimates across different 
      models. If the authors think it’s important, they could summarize the 
      inferred parameters for each model. Lastly, I noticed a few 
      typographical errors: L92: “…all participants played same, but…” 
      L128: “…inset shows example Beta prior…” L351: “…always the class 
      for…” L406: “…their ability of predicting…” L407: “In real world…”