  Prediction of pedestrian behavior is critical for fully autonomous vehicles
to drive in busy city streets safely and efficiently. The future autonomous
cars need to fit into mixed conditions with not only technical but also social
capabilities. As more algorithms and datasets have been developed to predict
pedestrian behaviors, these efforts lack the benchmark labels and the
capability to estimate the temporal-dynamic intent changes of the pedestrians,
provide explanations of the interaction scenes, and support algorithms with
social intelligence. This paper proposes and shares another benchmark dataset
called the IUPUI-CSRC Pedestrian Situated Intent (PSI) data with two innovative
labels besides comprehensive computer vision labels. The first novel label is
the dynamic intent changes for the pedestrians to cross in front of the
ego-vehicle, achieved from 24 drivers with diverse backgrounds. The second one
is the text-based explanations of the driver reasoning process when estimating
pedestrian intents and predicting their behaviors during the interaction
period. These innovative labels can enable several computer vision tasks,
including pedestrian intent/behavior prediction, vehicle-pedestrian interaction
segmentation, and video-to-language mapping for explainable algorithms. The
released dataset can fundamentally improve the development of pedestrian
behavior prediction models and develop socially intelligent autonomous cars to
interact with pedestrians efficiently. The dataset has been evaluated with
different tasks and is released to the public to access.

Computer Vision and Pattern Recognition;Artificial Intelligence