  We introduce the new setting of open-vocabulary object 6D pose estimation, in
which a textual prompt is used to specify the object of interest. In contrast
to existing approaches, in our setting (i) the object of interest is specified
solely through the textual prompt, (ii) no object model (e.g. CAD or video
sequence) is required at inference, (iii) the object is imaged from two
different viewpoints of two different scenes, and (iv) the object was not
observed during the training phase. To operate in this setting, we introduce a
novel approach that leverages a Vision-Language Model to segment the object of
interest from two distinct scenes and to estimate its relative 6D pose. The key
of our approach is a carefully devised strategy to fuse object-level
information provided by the prompt with local image features, resulting in a
feature space that can generalize to novel concepts. We validate our approach
on a new benchmark based on two popular datasets, REAL275 and Toyota-Light,
which collectively encompass 39 object instances appearing in four thousand
image pairs. The results demonstrate that our approach outperforms both a
well-established hand-crafted method and a recent deep learning-based baseline
in estimating the relative 6D pose of objects in different scenes. Project
page: https://jcorsetti.github.io/oryon/.

Computer Vision and Pattern Recognition