Learning under uncertainty: a comparison between R-W and Bayesian approach
Accurately differentiating between what are truly unpredictably random and systematic changes that occur at random can have profound effect on affect and cognition. To examine the underlying computational principles that guide different learning behavior in an uncertain environment, we compared an R-W model and a Bayesian approach in a visual search task with different volatility levels. Both R-W model and the Bayesian approach reflected an individual's estimation of the environmental volatility, and there is a strong correlation between the learning rate in R-W model and the belief of stationarity in the Bayesian approach in different volatility conditions. In a low volatility condition, R-W model indicates that learning rate positively correlates with lose-shift rate, but not choice optimality (inverted U shape). The Bayesian approach indicates that the belief of environmental stationarity positively correlates with choice optimality, but not lose-shift rate (inverted U shape). In addition, we showed that comparing to Expert learners, individuals with high lose-shift rate (sub-optimal learners) had significantly higher learning rate estimated from R-W model and lower belief of stationarity from the Bayesian model.
This paper compares an R-W model (with various constant learning rates) and a Bayesian model (with various constant beliefs of environmental stationarity). Section 2 verifies assumptions about the relation between the learning rate parameter and the performance and behavior of in-silico simulation runs of the R-W model, in three volatility conditions (section 2.1), and for three behavior types (section 2.2). Section 3 verifies assumptions about the relation between the belief of stationarity parameter and the performance and behavior of in-silico simulation runs of the Bayesian model, in three volatility conditions (section 3.1), and for three behavior types (section 3.2). Section 4 compares both models (R-W and Bayesian) "fitted" to simulate experiments performed by (Yu & Huang, 2014) with human subjects in a visual search task. 
You cite (Gershman 2015), but you do not provide an inductive analysis of your experimental results in such an unifying framework. Most of your analysis is deductive or consists in verifying that your simulation results match intuitive reasoning such as "learning rate is expected to increase as the volatility increases" (at line 137).  My main comments on the form are related to clarifications of your approach and methods: - A presentation of your simulation method could improve readability. Notably I guess that you have manipulated the R-W learning rate η (i.e. you ran the simulation with various learning rates) and the belief of stationarity α, in which case η and α are simulation inputs; this is not obvious at first reading of "Learning rate is expected to increase as the volatility increases", which suggests that the learning rate is an output of the simulation. - In section §4 "Experiment", whether η and α are inputs or outputs is less clear. Maybe this is due to the "fitting" process mentioned at line 268: in my understanding, fitting consists in running several simulations, setting (inputting) different parameter (η or α) values for each run, then selecting the parameter value which produces the best match with experimental data; hence a given parameter is both an input of each run and an output of the whole fitting process. Maybe a qualitative and a quantitative descriptions of such a fitting process (and how you select the best match) would clarify both your approach and your method. - You should define how you measure "model accuracy" (figure 3 and line 241), or maybe refer to the "Data Analysis: Model Predictive Accuracy" section of (Yu et al. 2014). - At line 268 you mention "subjects", I understand from (Yu et al. 2014) that they are human subjects, but you could recall this fact in your paper.  Below are detailed comments on the form: - The text is not aligned with line numbers. In my comments I refer to the line number just above the commented text line. - Line 40: you cite "(Mathys et al. 2011)", but the referenced paper is "(Mathys et al. 2014)". - The last paragraph of section 1 (lines 91-98) refers to two questions. Apparently, the first question starts at line 66, and the second question starts at line 86 ("In addition ..."). Maybe you could highlight the two questions in the next to last paragraph of section 1. - Line 132 "..., with a large β approximates ’maximization’ strategy.": "..., where a large β..."? - Line 160 (or 161) "... agents strictly follow WSLS have ...": "... agents which strictly follow WSLS have ..."? - Line 161 "... indicating their choices are heavily relied on recent experience": is "relied" the correct term here? Do you mean "linked"? Or "... recent experience heavily impacts their choices"? - Line 230-231 "with α = 0 leads to": same comment as line 132. - Line 238-239 "This is consistent with their estimation of higher volatility reﬂected by higher learning rate than Expert estimated from R-W model (Fig 2c).": I do not understand this sentence. - Line 268 "with change points occur": same comment as line 132. - Line 287 "inverse decision parameters": you define the R-W parameter β at line 132, but you do not define the "inverse decision parameter" used with your DBM simulations. Maybe you should refer to the corresponding parameter β of (Yu et al. 2014 p283-284). - Line 311 "In a visual search task with relative reward frequency 1:3:9 and change of the reward contingency occur based on N(30, 1)": quoting the settings, notably "change of the reward contingency occur based on N(30, 1)", may help readability. - Line 321: "Fig. 6c" should be replaced by "Fig. 5c". - Fig. 5c and lines 321-323: what is/are the subject categorie/s related to fig. 5c? What is/are the model results or human data related to fig. 5c? - Line 356 / Fig. 6a caption: what is "η0"? - Lines 424-425: You should not include acknowledgments in the submitted paper. - Line 370 "with a fixed learning rate and a Bayesian model with a fixed belief of stationarity": these formulation suggest a constant value for each parameter (see my comment above about fitting). You could formulate this way: "with various constant learning rates and a Bayesian model with various constant beliefs of stationarity"   Response to author rebuttals: Thanks for addressing many of my concerns. Considering your response to methods clarification requests, I have increased the "clarity" note. Considering other reviewers' qualitative assessment, I've increased the "technical" note. I am less confident about an improvement of novelty and impact criteria. As stated your response, your aim is rather experimental than theoretical.