Interfacing Foundation Models' Embeddings
We present FIND, a generalized interface for aligning foundation models'
embeddings. As shown in teaser figure, a lightweight transformer interface
without tuning any foundation model weights is enough for a unified image
(segmentation) and dataset-level (retrieval) understanding. The proposed
interface has the following favorable attributes: (1) Generalizable. It applies
to various tasks spanning retrieval, segmentation, etc., under the
same architecture and weights. (2) Prototypable. Different tasks are able to be
implemented through prototyping attention masks and embedding types. (3)
Extendable. The proposed interface is adaptive to new tasks, and new models.
(4) Interleavable. With the benefit of multi-task multi-modal training, the
proposed interface creates an interleaved shared embedding space. In light of
the interleaved embedding space, we introduce the FIND-Bench, which introduces
new training and evaluation annotations to the COCO dataset for interleave
segmentation and retrieval. Our approach achieves state-of-the-art performance
on FIND-Bench and competitive performance on standard retrieval and
segmentation settings. The training, evaluation, and demo code as well as the
dataset have been released at https://github.com/UX-Decoder/FIND.
6579427e62d3ac181719b219
Xueyan Zou
admin_assigned
2023-12-13T09:07:26.713Z
62520988818a5dc29ab91d6f
https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
Xueyan Zou
user
xueyanz
6579427e62d3ac181719b21a
Linjie Li
admin_assigned
2023-12-13T09:07:42.023Z
63db16fff03c3d71ef397206
/avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
Linjie Li
user
linjieli222
6579427e62d3ac181719b21b
Jianfeng Wang
admin_assigned
2023-12-13T09:08:04.225Z
641404e74e5305c14f269b3c
/avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
Jianfeng Wang
user
amsword8
6579427e62d3ac181719b21c
Jianwei Yang
admin_assigned
2023-12-13T09:17:32.774Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
6579427e62d3ac181719b21d
Mingyu Ding
6579427e62d3ac181719b21e
Zhengyuan Yang
admin_assigned
2023-12-13T09:17:47.980Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
6579427e62d3ac181719b21f
Feng Li
6579427e62d3ac181719b220
Hao Zhang
6579427e62d3ac181719b221
Shilong Liu
admin_assigned
2023-12-13T09:19:19.982Z
638b13c0c1d591879698f4e2
https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
Shilong Liu
user
ShilongLiu
6579427e62d3ac181719b222
Arul Aravinthan
admin_assigned
2023-12-13T09:19:26.921Z
64b6d1d733c80d14a2039cbf
/avatars/d68f4167c399962fb769d76b9fbc8c5f.svg
Arul Aravinthan
user
arul-aravinthan
6579427e62d3ac181719b223
Yong Jae Lee
admin_assigned
2023-12-13T09:19:36.401Z
649f41ee70a478f8b36b2984
/avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
Yong Jae Lee
user
yjlee0222
6579427e62d3ac181719b224
Lijuan Wang
admin_assigned
2023-12-13T09:19:42.621Z
6413521d4e5305c14f22e110
/avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
Lijuan Wang
user
Lijuan
Interfacing Foundation Models' Embeddings
We present FIND, a generalized interface for aligning foundation models'
embeddings. As shown in teaser figure, a lightweight transformer interface
without tuning any foundation model weights is enough for a unified image
(segmentation) and dataset-level (retrieval) understanding. The proposed
interface has the following favorable attributes: (1) Generalizable. It applies
to various tasks spanning retrieval, segmentation, etc., under the
same architecture and weights. (2) Prototypable. Different tasks are able to be
implemented through prototyping attention masks and embedding types. (3)
Extendable. The proposed interface is adaptive to new tasks, and new models.
(4) Interleavable. With the benefit of multi-task multi-modal training, the
proposed interface creates an interleaved shared embedding space. In light of
the interleaved embedding space, we introduce the FIND-Bench, which introduces
new training and evaluation annotations to the COCO dataset for interleave
segmentation and retrieval. Our approach achieves state-of-the-art performance
on FIND-Bench and competitive performance on standard retrieval and
segmentation settings. The training, evaluation, and demo code as well as the
dataset have been released at https://github.com/UX-Decoder/FIND.
FIND is a transformer interface that aligns foundation model embeddings for unified image segmentation and dataset-level retrieval without tuning the foundation models.
transformer foundation models embeddings unified understanding attention masks embedding types interleaved shared embedding space FIND-Bench COCO dataset interleave segmentation retrieval