  We propose and study a new computer vision task named open-vocabulary video
instance segmentation (OpenVIS), which aims to simultaneously segment, detect,
and track arbitrary objects in a video according to corresponding text
descriptions. Compared to the original video instance segmentation, OpenVIS
enables users to identify objects of desired categories, regardless of whether
those categories were included in the training dataset. To achieve this goal,
we propose a two-stage pipeline for proposing high-quality class-agnostic
object masks and predicting their corresponding categories via pre-trained VLM.
Specifically, we first employ a query-based mask proposal network to generate
masks of all potential objects, where we replace the original class head with
an instance head trained with a binary object loss, thereby enhancing the
class-agnostic mask proposal ability. Then, we introduce a proposal
post-processing approach to adapt the proposals better to the pre-trained VLMs,
avoiding distortion and unnatural proposal inputs. Meanwhile, to facilitate
research on this new task, we also propose an evaluation benchmark that
utilizes off-the-shelf datasets to comprehensively assess its performance.
Experimentally, the proposed OpenVIS exhibits a remarkable 148\% improvement
compared to the full-supervised baselines on BURST, which have been trained on
all categories.

Computer Vision and Pattern Recognition