In autonomous vehicle (AV) control, requiring an agent to make mistakes, or even allowing mistakes, can be quite dangerous and costly in the real world. For this reason we investigate methods of training an AV without allowing the agent to explore and instead having a human explorer collect the data. Supervised learning has been explored for AV control, but it encounters the issue of the covariate shift. That is, training data collected from an optimal demonstration consists only of the states induced by the optimal control policy, but at runtime, the trained agent may encounter a vastly different state distribution and has seen little relevant training data. To mitigate this issue, we intentionally have our human explorer make sub-optimal decisions. In order to have our agent not replicate these suboptimal decisions, supervised learning requires that we either erase these actions, or replace these action with the correct action. Erasing these actions is wasteful and replacing these actions is difficult, since it is not easy to know the correct action without driving the car. Since supervised learning falls short, we introduce an alternate framework that includes continuous scalar feedback on each action, marking which actions we should replicate, which we should avoid, and how sure of this we are. Our framework, called "ReNeg", learns from from human demonstration and human evaluative feedback, collected entirely before any training begins. Our agent learns continuous control from sub-optimal behavior, but sub-optimal behavior that is safely performed by a human. We find that a human demonstrator can explore sub-optimal states in a safe and controlled manner, while still getting enough gradation in the states to benefit learning. The way we collect data and the algorithm we use to compute feedback on the data we call "Backseat Driver." Backseat Driver gives us state-action pairs matched with scalar values representing the score for those action in those states. We call the more general learning framework ReNeg, since it learns a regression from states to actions given negative as well as positive examples. We empirically validate several models in the ReNeg framework, testing on lane-following with limited data. We find that the best solution is a generalization of mean-squared error and outperforms supervised learning on the positive examples alone.
Despite many high profile successes in research, DeepRL is still not widely used in applications. One reason for this is that current methods typically assume the agent can learn by exploring many states and actions, however, in many real world tasks, such as driving used here, poor actions can be dangerous. Therefore, methods that can provides the flexible benefits of RL while avoiding this are of significant interest, one promising general ideas pursued for this has been to use human demonstrations.
A number of approaches to Inverse RL have been studied, but many make the assumption that the demonstrations are all examples of optimal behavior. This can be challenging if, for example, some examples contain suboptimal behavior, and it also means that the agent does not get to observe non-optimal trajectories and how to correct for them; the resulting policy often performs poorly due to the distributional shift between the demonstration trajectories and the trajectories induced by the learned policy.
This work attempts to correct for these problems by labeling the demonstration actions between $[-1, 1]$ indicating how good or bad the demonstration actions are. This introduces a challenge for learning, since good actions can be copied, but a bad action is more ambiguous: it does not necessarily imply the action are far away from the bad action is a good action.
One view of this work is that they introducing 3 losses for behavior cloning with weighted labels: A weighted (based on the label) L2 loss, an exponential loss and directly fitting the loss and searching over a discrete set of actions to find the highest estimate weighting. Note the current equation for $Loss_{FNET}$ doesn't make sense because it simply minimizing the output of the network, from the text it should be something like $(f - \hat{\theta})^2$?
The text discusses why rescaling the negative examples may be beneficial, but as far as I can tell, figure 4 you only consider $\alpha=\{0, 1\}$? Based on the text, why weren't intermediate values of $\alpha$ considered?
It could benefit from copy-editing, checking the equations and in some cases describing concepts more concisely using clear mathematical notation instead of wordy descriptions that are difficult to follow.
``Thus the assumption
that our training data is independent and identically distributed (i.i.d.) from the agent’s encountered
distribution goes out the window'' This is a misleading statement regarding the challenge of distributional shift in off-policy RL. The challenge is that state distribution between the behavior policy and the learned policy may be quite different, not just not iid. Even in on-policy RL the state visitation is certainly not usually iid.
``In the off-policy policy gradient RL framework, this issue is typically circumvented by changing the
objective function from an expectation of the learned policy’s value function over the learned policy
state-visitation distribution to an expectation of the learned policy’s value function over the behavior
(exploratory) state-visitation distribution (Degris et al., 2012). In the RL framework, this could be
dealt with by an approximation off-policy stochastic policy gradient that scales the stochastic policy
gradient by an importance sampling ratio (Silver et al., 2014) (Degris et al., 2012). ''. The importance sampling in Degris is not to correct for the objective being under the behavior state policy and DPG (Silver et al., 2014) specifically does not require importance sampling so it shouldn't be referenced here. This paragraph seems to be conflating two issues: the distributional shift between the behavior state distribution and the policy state distribution that can make off-policy learning unstable, and importance sampling to estimate outcome likelihoods using behavior experience.
This work is on a very important topic. However, in its current form it is not well-communicated. Additionally, the best performing method is not novel (as the author's state $\alpha=1$, the best performing setting, is essentially the same as COACH but with scalar labels). For these reasons reason, I think this work may be of limited interested.