title: Segment and Caption Anything
summary: We propose a method to efficiently equip the Segment Anything Model (SAM)
  with the ability to generate regional captions. SAM presents strong
  generalizability to segment anything while is short for semantic understanding.
  By introducing a lightweight query-based feature mixer, we align the
  region-specific features with the embedding space of language models for later
  caption generation. As the number of trainable parameters is small (typically
  in the order of tens of millions), it costs less computation, less memory
  usage, and less communication bandwidth, resulting in both fast and scalable
  training. To address the scarcity problem of regional caption data, we propose
  to first pre-train our model on objection detection and segmentation tasks. We
  call this step weak supervision pretraining since the pre-training data only
  contains category names instead of full-sentence descriptions. The weak
  supervision pretraining allows us to leverage many publicly available object
  detection and segmentation datasets. We conduct extensive experiments to
  demonstrate the superiority of our method and validate each design choice. This
  work serves as a stepping stone towards scaling up regional captioning data and
  sheds light on exploring efficient ways to augment SAM with regional semantics.
  The project page, along with the associated code, can be accessed via the
  following https://xk-huang.github.io/segment-caption-anything/.
paper_authors._id: 656e960cc94d150e5de4994a
paper_authors.name: Xiaoke Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:56:36.086Z
paper_authors._id: 63318b2349a9563915469f3b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63318b2349a9563915469f3b/zlbeB2997i8YkoyOTb9FL.jpeg
paper_authors.fullname: Xiaoke Huang
paper_authors.type: user
paper_authors.user: xk-huang
paper_authors._id: 656e960cc94d150e5de4994b
paper_authors.name: Jianfeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:56:59.745Z
paper_authors._id: 641404e74e5305c14f269b3c
paper_authors.avatarUrl: /avatars/0dbe862cde7e4bef25bedf860584d8c1.svg
paper_authors.fullname: Jianfeng Wang
paper_authors.type: user
paper_authors.user: amsword8
paper_authors._id: 656e960cc94d150e5de4994c
paper_authors.name: Yansong Tang
paper_authors._id: 656e960cc94d150e5de4994d
paper_authors.name: Zheng Zhang
paper_authors._id: 656e960cc94d150e5de4994e
paper_authors.name: Han Hu
paper_authors._id: 656e960cc94d150e5de4994f
paper_authors.name: Jiwen Lu
paper_authors._id: 656e960cc94d150e5de49950
paper_authors.name: Lijuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:58:44.759Z
paper_authors._id: 6413521d4e5305c14f22e110
paper_authors.avatarUrl: /avatars/a6f8d0573e678f79bc3c0b7897b818ce.svg
paper_authors.fullname: Lijuan Wang
paper_authors.type: user
paper_authors.user: Lijuan
paper_authors._id: 656e960cc94d150e5de49951
paper_authors.name: Zicheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T08:58:53.583Z
paper_authors._id: 6569f6cb44ce94a70187f407
paper_authors.avatarUrl: /avatars/414ecc9739ed23f9f395bd7cfa65055f.svg
paper_authors.fullname: Zicheng Liu
paper_authors.type: user
paper_authors.user: ZCLiu35
paper_title: Segment and Caption Anything
paper_summary: We propose a method to efficiently equip the Segment Anything Model (SAM)
  with the ability to generate regional captions. SAM presents strong
  generalizability to segment anything while is short for semantic understanding.
  By introducing a lightweight query-based feature mixer, we align the
  region-specific features with the embedding space of language models for later
  caption generation. As the number of trainable parameters is small (typically
  in the order of tens of millions), it costs less computation, less memory
  usage, and less communication bandwidth, resulting in both fast and scalable
  training. To address the scarcity problem of regional caption data, we propose
  to first pre-train our model on objection detection and segmentation tasks. We
  call this step weak supervision pretraining since the pre-training data only
  contains category names instead of full-sentence descriptions. The weak
  supervision pretraining allows us to leverage many publicly available object
  detection and segmentation datasets. We conduct extensive experiments to
  demonstrate the superiority of our method and validate each design choice. This
  work serves as a stepping stone towards scaling up regional captioning data and
  sheds light on exploring efficient ways to augment SAM with regional semantics.
  The project page, along with the associated code, can be accessed via the
  following https://xk-huang.github.io/segment-caption-anything/.
paper_ai_summary: A method enhances the Segment Anything Model (SAM) with regional caption generation using a lightweight query-based feature mixer, enabling efficient use of weak supervision pretraining on publicly available datasets.
paper_ai_keywords: Segment Anything Model SAM regional captions query-based feature mixer embedding space language models weak supervision pretraining object detection segmentation regional semantics