title: Descriptive Caption Enhancement with Visual Specialists for Multimodal
    Perception
summary: Training Large Multimodality Models (LMMs) relies on descriptive image
  caption that connects image and language. Existing methods either distill the
  caption from the LMM models or construct the captions from the internet images
  or by human. We propose to leverage off-the-shelf visual specialists, which
  were trained from annotated images initially not for image captioning, for
  enhancing the image caption.
    Our approach, named DCE, explores object low-level and fine-grained
  attributes (e.g., depth, emotion and fine-grained categories) and object
  relations (e.g., relative location and human-object-interaction (HOI)), and
  combine the attributes into the descriptive caption. Experiments demonstrate
  that such visual specialists are able to improve the performance for visual
  understanding tasks as well as reasoning that benefits from more accurate
  visual understanding. We will release the source code and the pipeline so that
  other visual specialists are easily combined into the pipeline. The complete
  source code of DCE pipeline and datasets will be available at
  https://github.com/syp2ysy/DCE.
paper_authors._id: 6764e3c22086097d58dc7fc4
paper_authors.name: Yanpeng Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:32:42.316Z
paper_authors._id: 64297212e5f33939cf3a3d9b
paper_authors.avatarUrl: /avatars/bd21759ab5d7e526b99fcb7ed813ffb3.svg
paper_authors.fullname: yanpeng_sun
paper_authors.type: user
paper_authors.user: syp115
paper_authors._id: 6764e3c22086097d58dc7fc5
paper_authors.name: Jing Hao
paper_authors._id: 6764e3c22086097d58dc7fc6
paper_authors.name: Ke Zhu
paper_authors._id: 6764e3c22086097d58dc7fc7
paper_authors.name: Jiang-Jiang Liu
paper_authors._id: 6764e3c22086097d58dc7fc8
paper_authors.name: Yuxiang Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:33:04.173Z
paper_authors._id: 641966f6e8a6183a8caa3145
paper_authors.avatarUrl: /avatars/a7d7096fed7e49fcc04f2ef494e9c381.svg
paper_authors.fullname: yuxiang zhao
paper_authors.type: user
paper_authors.user: cloud913
paper_authors._id: 6764e3c22086097d58dc7fc9
paper_authors.name: Xiaofan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:33:18.741Z
paper_authors._id: 65f425f9bfb2f1451f237827
paper_authors.avatarUrl: /avatars/36b0bce9364d88f10081847befd29787.svg
paper_authors.fullname: Xiaofan Li
paper_authors.type: user
paper_authors.user: FuNz
paper_authors._id: 6764e3c22086097d58dc7fca
paper_authors.name: Gang Zhang
paper_authors._id: 6764e3c22086097d58dc7fcb
paper_authors.name: Zechao Li
paper_authors._id: 6764e3c22086097d58dc7fcc
paper_authors.name: Jingdong Wang
paper_title: Descriptive Caption Enhancement with Visual Specialists for Multimodal
    Perception
paper_summary: Training Large Multimodality Models (LMMs) relies on descriptive image
  caption that connects image and language. Existing methods either distill the
  caption from the LMM models or construct the captions from the internet images
  or by human. We propose to leverage off-the-shelf visual specialists, which
  were trained from annotated images initially not for image captioning, for
  enhancing the image caption.
    Our approach, named DCE, explores object low-level and fine-grained
  attributes (e.g., depth, emotion and fine-grained categories) and object
  relations (e.g., relative location and human-object-interaction (HOI)), and
  combine the attributes into the descriptive caption. Experiments demonstrate
  that such visual specialists are able to improve the performance for visual
  understanding tasks as well as reasoning that benefits from more accurate
  visual understanding. We will release the source code and the pipeline so that
  other visual specialists are easily combined into the pipeline. The complete
  source code of DCE pipeline and datasets will be available at
  https://github.com/syp2ysy/DCE.
paper_ai_summary: DCE enhances image captions by integrating attributes and relationships from pretrained visual specialists, improving visual understanding and reasoning tasks.
paper_ai_keywords: multimodality models image captioning visual specialists object attributes fine-grained attributes object relations human-object interaction visual understanding reasoning