title: Interfacing Foundation Models' Embeddings
summary: We present FIND, a generalized interface for aligning foundation models'
  embeddings. As shown in teaser figure, a lightweight transformer interface
  without tuning any foundation model weights is enough for a unified image
  (segmentation) and dataset-level (retrieval) understanding. The proposed
  interface has the following favorable attributes: (1) Generalizable. It applies
  to various tasks spanning retrieval, segmentation, etc., under the
  same architecture and weights. (2) Prototypable. Different tasks are able to be
  implemented through prototyping attention masks and embedding types. (3)
  Extendable. The proposed interface is adaptive to new tasks, and new models.
  (4) Interleavable. With the benefit of multi-task multi-modal training, the
  proposed interface creates an interleaved shared embedding space. In light of
  the interleaved embedding space, we introduce the FIND-Bench, which introduces
  new training and evaluation annotations to the COCO dataset for interleave
  segmentation and retrieval. Our approach achieves state-of-the-art performance
  on FIND-Bench and competitive performance on standard retrieval and
  segmentation settings. The training, evaluation, and demo code as well as the
  dataset have been released at https://github.com/UX-Decoder/FIND.
paper_authors._id: 6579427e62d3ac181719b219
paper_authors.name: Xueyan Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:07:26.713Z
paper_authors._id: 62520988818a5dc29ab91d6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
paper_authors.fullname: Xueyan Zou
paper_authors.type: user
paper_authors.user: xueyanz
paper_authors._id: 6579427e62d3ac181719b21a
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:07:42.023Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 6579427e62d3ac181719b21b
paper_authors.name: Jianfeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:08:04.225Z
paper_authors._id: 641404e74e5305c14f269b3c
paper_authors.avatarUrl: /avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
paper_authors.fullname: Jianfeng Wang
paper_authors.type: user
paper_authors.user: amsword8
paper_authors._id: 6579427e62d3ac181719b21c
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:17:32.774Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 6579427e62d3ac181719b21d
paper_authors.name: Mingyu Ding
paper_authors._id: 6579427e62d3ac181719b21e
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:17:47.980Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 6579427e62d3ac181719b21f
paper_authors.name: Feng Li
paper_authors._id: 6579427e62d3ac181719b220
paper_authors.name: Hao Zhang
paper_authors._id: 6579427e62d3ac181719b221
paper_authors.name: Shilong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:19:19.982Z
paper_authors._id: 638b13c0c1d591879698f4e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
paper_authors.fullname: Shilong Liu
paper_authors.type: user
paper_authors.user: ShilongLiu
paper_authors._id: 6579427e62d3ac181719b222
paper_authors.name: Arul Aravinthan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:19:26.921Z
paper_authors._id: 64b6d1d733c80d14a2039cbf
paper_authors.avatarUrl: /avatars/d68f4167c399962fb769d76b9fbc8c5f.svg
paper_authors.fullname: Arul Aravinthan
paper_authors.type: user
paper_authors.user: arul-aravinthan
paper_authors._id: 6579427e62d3ac181719b223
paper_authors.name: Yong Jae Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:19:36.401Z
paper_authors._id: 649f41ee70a478f8b36b2984
paper_authors.avatarUrl: /avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
paper_authors.fullname: Yong Jae Lee
paper_authors.type: user
paper_authors.user: yjlee0222
paper_authors._id: 6579427e62d3ac181719b224
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T09:19:42.621Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_title: Interfacing Foundation Models' Embeddings
paper_summary: We present FIND, a generalized interface for aligning foundation models'
  embeddings. As shown in teaser figure, a lightweight transformer interface
  without tuning any foundation model weights is enough for a unified image
  (segmentation) and dataset-level (retrieval) understanding. The proposed
  interface has the following favorable attributes: (1) Generalizable. It applies
  to various tasks spanning retrieval, segmentation, etc., under the
  same architecture and weights. (2) Prototypable. Different tasks are able to be
  implemented through prototyping attention masks and embedding types. (3)
  Extendable. The proposed interface is adaptive to new tasks, and new models.
  (4) Interleavable. With the benefit of multi-task multi-modal training, the
  proposed interface creates an interleaved shared embedding space. In light of
  the interleaved embedding space, we introduce the FIND-Bench, which introduces
  new training and evaluation annotations to the COCO dataset for interleave
  segmentation and retrieval. Our approach achieves state-of-the-art performance
  on FIND-Bench and competitive performance on standard retrieval and
  segmentation settings. The training, evaluation, and demo code as well as the
  dataset have been released at https://github.com/UX-Decoder/FIND.
paper_ai_keywords: transformer interface embeddings foundation models unified understanding image segmentation dataset-level retrieval attention masks embedding types multi-task multi-modal training interleaved shared embedding space FIND-Bench COCO dataset interleave segmentation retrieval settings