Embodiments are disclosed for scoring one or more trajectories of a vehicle through a given traffic scenario using a machine learning model that predicts rationality scores for the trajectories. In an embodiment, a rendering of two or more vehicle trajectories through the same or different traffic scenarios is presented to a human annotator referred to as a 'reasonable population'. The annotators are required to indicate that the annotators preference one trajectory rather than other trajectory (s). A machine learning model is trained using inputs collected from human annotators to predict rationality scores for one or more trajectories for a given traffic scene. The predicted trajectories may be used to rank the trajectories based on scores of the trajectories generated by a route planner, compare AV software stacks, or may be used by any other application that may benefit from machine learning models that score vehicle trajectories.
Scoring autonomous vehicle trajectory using reasonable crowd data