  Prompt tuning with large-scale pretrained vision-language models empowers
open-vocabulary predictions trained on limited base categories, e.g., object
classification and detection. In this paper, we propose compositional prompt
tuning with motion cues: an extended prompt tuning paradigm for compositional
predictions of video data. In particular, we present Relation Prompt (RePro)
for Open-vocabulary Video Visual Relation Detection (Open-VidVRD), where
conventional prompt tuning is easily biased to certain subject-object
combinations and motion patterns. To this end, RePro addresses the two
technical challenges of Open-VidVRD: 1) the prompt tokens should respect the
two different semantic roles of subject and object, and 2) the tuning should
account for the diverse spatio-temporal motion patterns of the subject-object
compositions. Without bells and whistles, our RePro achieves a new
state-of-the-art performance on two VidVRD benchmarks of not only the base
training object and predicate categories, but also the unseen ones. Extensive
ablations also demonstrate the effectiveness of the proposed compositional and
multi-mode design of prompts. Code is available at
https://github.com/Dawn-LX/OpenVoc-VidVRD.

Computer Vision and Pattern Recognition