Not All Errors Are Made Equal: A Regret Metric for Detecting System-level Trajectory Prediction Failures
<|reference_start|>Not All Errors Are Made Equal: A Regret Metric for Detecting System-level Trajectory Prediction Failures: Robot decision-making increasingly relies on data-driven human prediction models when operating around people. While these models are known to mispredict in out-of-distribution interactions, only a subset of prediction errors impact downstream robot performance. We propose characterizing such "system-level" prediction failures via the mathematical notion of regret: high-regret interactions are precisely those in which mispredictions degraded closed-loop robot performance. We further introduce a probabilistic generalization of regret that calibrates failure detection across disparate deployment contexts and renders regret compatible with reward-based and reward-free (e.g., generative) planners. In simulated autonomous driving interactions and social navigation interactions deployed on hardware, we showcase that our system-level failure metric can be used offline to automatically extract closed-loop human-robot interactions that state-of-the-art generative human predictors and robot planners previously struggled with. We further find that the very presence of high-regret data during human predictor fine-tuning is highly predictive of robot re-deployment performance improvements. Fine-tuning with the informative but significantly smaller high-regret data (23% of deployment data) is competitive with fine-tuning on the full deployment dataset, indicating a promising avenue for efficiently mitigating system-level human-robot interaction failures. Project website: https://cmu-intentlab.github.io/not-all-errors/<|reference_end|>
@article{nakamura2024not,
    title={Not All Errors Are Made Equal: A Regret Metric for Detecting
  System-level Trajectory Prediction Failures},
    author={Kensuke Nakamura, Ran Tian, Andrea Bajcsy},
    journal={arXiv preprint arXiv:2403.04745},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.04745},
    primaryClass={cs.RO}
}