Optimal decision-making with time-varying evidence reliability
Previous theoretical and experimental work on optimal decision-making was restricted to the artificial setting of a reliability of the momentary sensory evidence that remained constant within single trials. The work presented here describes the computation and characterization of optimal decision-making in the more realistic case of an evidence reliability that varies across time even within a trial. It shows that, in this case, the optimal behavior is determined by a bound in the decision maker's belief that depends only on the current, but not the past, reliability. We furthermore demonstrate that simpler heuristics fail to match the optimal performance for certain characteristics of the process that determines the time-course of this reliability, causing a drop in reward rate by more than 50%.
Optimal decision-making with time-varying evidence reliabilityIn this article, the authors formulate an ideal observer model for making a decision based on accumulated evidence over time, where the evidence reliability varies over time.  They assume the reliability of the evidence follows a stochastic process (a Cox-Ingersoll-Ross process) and that there is a cost for waiting to act that is linear in time. The resulting computational model is complex, and so they apply Dynamic Programming and Partial Differential Equation solvers to approximate the model given some data. They then explore the behavior of their model for different scenarios (e.g., increasing/decreasing the standard deviation of the reliability) and outline some intuitions for how the parameters affect model behavior. Finally, they compare model behavior to some simple heuristics to outline why those heuristics fail when evidence reliability is actually variable.The article tackles a difficult problem that is of interest to many within the NIPS community and makes theoretical advances, although they may be incremental advances. It uses mathematical techniques that are not currently prominent within the NIPS community (e.g., applying partial differential equation solvers to stochastic processes), which could have a positive impact on other areas with equivalent mathematical models. As a cognitive scientist with a moderate background in stochastic process theory, the mathematics in the article are beyond my ability to evaluate properly (beyond cursory reasonableness and consistency). So I will leave the critique of the mathematics to other reviewers. From this perspective, I found the article difficult to follow at times, not because it is poorly written (it is actually fairly well written), but because it is dense and the mathematics are difficult. I would have benefited from some more thorough explanations of the underlying assumptions (why a Cox-Ingersoll-Ross process as opposed to other continuous stochastic processes?) and some more thorough explanations of the figures (which are quite dense). For future work, it might be useful for the authors to explore using a more discontinuous stochastic process for the evidence reliability. Intuitively, it seems like evidence reliability is likely to stay constant except for when there is a change in the environment (in which case there would be a large jump). This is not meant as a criticism (the authors’ approach seems reasonable and to the best of my knowledge, the first of its kind), but rather as a suggestion for future work.Also, the article feels incomplete because they setup and develop a nice formalism, but never test it or compare it to previous empirical data. However, there is more than enough theoretical work in the article to justify its worth, and so it reflects my own bias. In fact, presenting the article at NIPS could result in useful collaborations to test the model.Minor comments:“stead” -> “steady” (in many places, it says “stead” state rather than “steady” state)