VISCERAL MACHINES: RISK-AVERSION IN REINFORCEMENT LEARNING WITH INTRINSIC PHYSIOLOGICAL REWARDS
As people learn to navigate the world, autonomic nervous system (e.g., "fight or flight") responses provide intrinsic feedback about the potential consequence of action choices (e.g., becoming nervous when close to a cliff edge or driving fast around a bend.) Physiological changes are correlated with these biological preparations to protect one-self from danger. We present a novel approach to reinforcement learning that leverages a task-independent intrinsic reward function trained on peripheral pulse measurements that are correlated with human autonomic nervous system responses. Our hypothesis is that such reward functions can circumvent the challenges associated with sparse and skewed rewards in reinforcement learning settings and can help improve sample efficiency. We test this in a simulated driving environment and show that it can increase the speed of learning and reduce the number of collisions during the learning stage.
Summary:
This submission proposes a reinforcement learning framework based on human emotional reaction in the context of autonomous driving. This relies on defining a reward function as the convex combination of an extrinsic (goal oriented) reward, and an intrinsic reward. This later reward is learnt from experiments with humans performing the task in a virtual environment, for which emotional response is quantified as blood volume pulse wave (BVP). The authors show that including this intrinsic reward lead to a better performance of a deep Q networks, with respect to using the extrinsic reward only. 
Evaluation:
Overall the proposed idea is interesting, and the use of human experiments to improve a reinforcement learning algorithm offers interesting perspectives. The weakness of the paper in my opinion is the statistical analysis of the results, the lack of in depth evaluation of the extrinsic reward prediction and the rather poor baseline comparison.
Detailed comments:
1.	Statistical analysis
The significance of the results should be assessed with statistical methods in the following results:
Section 4.1: Please provide and assessment of the significance of the testing loss of the prediction. For example, one could repetitively shuffle blocks of the target time series and quantify the RMSE obtained by the trained algorithm to build an H0 statistic of random prediction.
Section 4.2: the sentence “improves significantly when lambda is either non-zero or not equal to 1” does not seem valid to me and such claim should in any case be properly evaluated statistically (including correction for multiple comparison etc…).
Error bars: please provide a clear description in the figure caption of what the error bars represent. Ideally in case of small samples, box plots would be more appropriate.
2.	Time lags in BVP
It would be interesting to know (from the literature) the typical latency of BVP responses to averse stimuli (and possible the latency of the various mechanisms, e.g. brain response, in the chain from stimuli to BVP). Moreover, as latency is likely a critical factor in anticipating danger before it is too late, it would important to know how the prediction accuracy evolves when learning to predict at different time lags forward in time, and how such level of anticipation influence the performance of the Q-network.
3.	Poor baseline comparison
The comparison to reward shaping in section 4.4 is not very convincing. One can imagine that what counts is not the absolute distance to a wall, but the distance to a wall in the driving direction, within a given solid angle. As a consequence, a better heuristic baseline could be used. 
Moreover, it is unclear whether the approaches should be compared with the same lambda: the authors need to provide evidence that the statistics (mean and possibly variance) of the chosen heuristic is match to the original intrinsic reward, otherwise it is obvious that the lambda should be adapted.
4.	Better analysis of figure 5-6(Minor)
I find figure 5-6 very interesting and I would suggest that the authors fully comment on these results. E.g. : (1) why the middle plot of Fig. 6 mostly flat, and why such differences between each curve from the beginning of the training. (2) Why the goal oriented task leads to different optimal lambda, is this just a normalization issue?