We consider the following central question in the field of Deep Reinforcement Learning (DRL): How can we use implicit human feedback to accelerate and optimize the training of a DRL algorithm? State-of-the-art methods rely on any human feedback to be provided explicitly, requiring the active participation of humans (e.g., expert labeling, demonstrations, etc.). In this work, we investigate an alternative paradigm, where non-expert humans are silently observing (and assessing) the agent interacting with the environment. The human's intrinsic reactions to the agent's behavior is sensed as implicit feedback by placing electrodes on the human scalp and monitoring what are known as event-related electric potentials. The implicit feedback is then used to augment the agent's learning in the RL tasks. We develop a system to obtain and accurately decode the implicit human feedback (specifically error-related event potentials) for state-action pairs in an Atari-type environment. As a baseline contribution, we demonstrate the feasibility of capturing error-potentials of a human observer watching an agent learning to play several different Atari-games using an electroencephalogram (EEG) cap, and then decoding the signals appropriately and using them as an auxiliary reward function to a DRL algorithm with the intent of accelerating its learning of the game. Building atop the baseline, we then make the following novel contributions in our work: (i) We argue that the definition of error-potentials is generalizable across different environments; specifically we show that error-potentials of an observer can be learned for a specific game, and the definition used as-is for another game without requiring re-learning of the error-potentials. (ii) We propose two different frameworks to combine recent advances in DRL into the error-potential based feedback system in a sample-efficient manner, allowing humans to provide implicit feedback while training in the loop, or prior to the training of the RL agent. (iii) Finally, we scale the implicit human feedback (via ErrP) based RL to reasonably complex environments (games) and demonstrate the significance of our approach through synthetic and real user experiments.
This paper tackles the problem of obtaining feedback from humans for training RL agents, in a way that does not require too much time or mental burden. The proposed approach does this by using implicit human feedback, obtained by measuring EEG signals from the brains of human observers. When humans spot an error (e.g., in the behaviour of the RL agent), an error potential (ErrP) is generated, which differs from human to human. The proposed approach first uses supervised learning to learn what the ErrP looks like for a particular person, and then uses this to provide an additional reward signal to the agent -- the agent receives a negative reward for state-action pairs that result in an ErrP.
The goal of this paper is to show that using implicit human feedback via ErrPs is a feasible way to speed up the training of RL agents. Towards this, the paper makes two main contributions: (1) experimental evidence that the ErrP for a human can be learned for one game, and transferred zero-shot to other games, and (2) two approaches of collecting this implicit feedback from humans for a smaller number of trajectories, instead of collecting it for all trajectories encountered by the agent during learning. The experimental evaluations, on three simple Atari-like domains, show that agents learn more quickly with implicit feedback, even when gathered for a small number of trajectories, compared to not having any feedback.
This paper is well-motivated, and tackles an important problem. However, I have several concerns related to feasibility and reproducibility, as described below. The writing also needs quite a bit of editing; there are typos, grammatical errors, and incorrect / missing figure and section references.
Regarding feasibility, my first concern is that the domains considered here are relatively simple. Is there evidence that learned ErrPs would also generalize between tasks with more complex visuals? Also, in order to obtain these error signals, agent trajectories need to be slowed down significantly (from multiple actions a second to one action every 1.5 seconds). I imagine this would quickly become very tedious for people. This seems to be an inherent limitation of this approach, because relevant EEG signals happen up to 1+ seconds after an event occurs. I have doubts that this would generalize to more complex, long-horizon tasks, in which trajectories are at least hundreds of timesteps long. Finally, the study was conducted on a small number of participants (five), with a narrow range of ages and limited gender diversity. I sympathize with the difficulty in obtaining subjects for such studies, but the small sample size makes it questionable whether these results on transferability of learned ErrPs would apply to the wider population.
Reproducibility questions:
- How are the labels (of error vs. not error) obtained for learning per-person error potentials? For more complex domains, it seems that it would be harder to obtain these ground-truth labels, because if there are substantially different strategies for playing the game, humans could disagree on which state-action pairs they consider to be errors.
- In Section 4.2, how exactly does the RL agent use the saved state-action error labels (from queries to the human) to determine if the current state-action pair is an error? Is it using some form of nearest-neighbor? The notion of buckets isn't clearly described.
Additional questions and comments:
- Why does Wobble have the worst AUC for learning ErrPs (Figure 5a)? It seems to be the game that has the most obvious errors (i.e., agent moves away from the target), so I would expect it to have the best AUC.
- In Section 5.2.1 (Generalizability), what are hypotheses for *why* the ErrP trained on Wobble generalizes better to certain situations in Catch and worse for others? It would be useful to be able to characterise / predict how good generalization will be from one type of game / situation to another.
- I would like to see reports of wall-clock time for collection of implicit human feedback, for full access and the two other approaches. In other words, how long do humans have to spend connected to the electrodes?
- I'm curious how imperfect the demonstrations need to be, in order for the ErrPs for those demonstrations to provide a useful reward signal to the agent. In the paper, incorrect actions appear with a probability of 0.2, but there's no explanation for how this number was selected, and whether others were tested.
Minor comments:
- The diagram in Figure 2a is confusing because DQN is there twice —- explicitly as a component, and implicitly as part of the RL agent.
- The x-axis labels in Figure 5b don't agree with the subject labels in Figures 4b and 7b.
- In Figure 6, data for only two of the five participants is reported; it would be useful to include the results for the other participants in the Appendix.