Descriptive Caption Enhancement with Visual Specialists for Multimodal
  Perception
Training Large Multimodality Models (LMMs) relies on descriptive image
caption that connects image and language. Existing methods either distill the
caption from the LMM models or construct the captions from the internet images
or by human. We propose to leverage off-the-shelf visual specialists, which
were trained from annotated images initially not for image captioning, for
enhancing the image caption.
  Our approach, named DCE, explores object low-level and fine-grained
attributes (e.g., depth, emotion and fine-grained categories) and object
relations (e.g., relative location and human-object-interaction (HOI)), and
combine the attributes into the descriptive caption. Experiments demonstrate
that such visual specialists are able to improve the performance for visual
understanding tasks as well as reasoning that benefits from more accurate
visual understanding. We will release the source code and the pipeline so that
other visual specialists are easily combined into the pipeline. The complete
source code of DCE pipeline and datasets will be available at
https://github.com/syp2ysy/DCE.
6764e3c22086097d58dc7fc4
Yanpeng Sun
claimed_verified
2024-12-20T08:32:42.316Z
64297212e5f33939cf3a3d9b
/avatars/bd21759ab5d7e526b99fcb7ed813ffb3.svg
yanpeng_sun
user
syp115
6764e3c22086097d58dc7fc5
Jing Hao
6764e3c22086097d58dc7fc6
Ke Zhu
6764e3c22086097d58dc7fc7
Jiang-Jiang Liu
6764e3c22086097d58dc7fc8
Yuxiang Zhao
admin_assigned
2024-12-20T09:33:04.173Z
641966f6e8a6183a8caa3145
/avatars/a7d7096fed7e49fcc04f2ef494e9c381.svg
yuxiang zhao
user
cloud913
6764e3c22086097d58dc7fc9
Xiaofan Li
admin_assigned
2024-12-20T09:33:18.741Z
65f425f9bfb2f1451f237827
/avatars/36b0bce9364d88f10081847befd29787.svg
Xiaofan Li
user
FuNz
6764e3c22086097d58dc7fca
Gang Zhang
6764e3c22086097d58dc7fcb
Zechao Li
6764e3c22086097d58dc7fcc
Jingdong Wang
Descriptive Caption Enhancement with Visual Specialists for Multimodal
  Perception
Training Large Multimodality Models (LMMs) relies on descriptive image
caption that connects image and language. Existing methods either distill the
caption from the LMM models or construct the captions from the internet images
or by human. We propose to leverage off-the-shelf visual specialists, which
were trained from annotated images initially not for image captioning, for
enhancing the image caption.
  Our approach, named DCE, explores object low-level and fine-grained
attributes (e.g., depth, emotion and fine-grained categories) and object
relations (e.g., relative location and human-object-interaction (HOI)), and
combine the attributes into the descriptive caption. Experiments demonstrate
that such visual specialists are able to improve the performance for visual
understanding tasks as well as reasoning that benefits from more accurate
visual understanding. We will release the source code and the pipeline so that
other visual specialists are easily combined into the pipeline. The complete
source code of DCE pipeline and datasets will be available at
https://github.com/syp2ysy/DCE.
DCE enhances image captions by integrating attributes and relationships from pretrained visual specialists, improving visual understanding and reasoning tasks.
multimodality models image captioning visual specialists object attributes fine-grained attributes object relations human-object interaction visual understanding reasoning