Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding
<|reference_start|>Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding: Panoptic narrative grounding (PNG), whose core target is fine-grained image-text alignment, requires a panoptic segmentation of referred objects given a narrative caption. Previous discriminative methods achieve only weak or coarse-grained alignment by panoptic segmentation pretraining or CLIP model adaptation. Given the recent progress of text-to-image Diffusion models, several works have shown their capability to achieve fine-grained image-text alignment through cross-attention maps and improved general segmentation performance. However, the direct use of phrase features as static prompts to apply frozen Diffusion models to the PNG task still suffers from a large task gap and insufficient vision-language interaction, yielding inferior performance. Therefore, we propose an Extractive-Injective Phrase Adapter (EIPA) bypass within the Diffusion UNet to dynamically update phrase prompts with image features and inject the multimodal cues back, which leverages the fine-grained image-text alignment capability of Diffusion models more sufficiently. In addition, we also design a Multi-Level Mutual Aggregation (MLMA) module to reciprocally fuse multi-level image and phrase features for segmentation refinement. Extensive experiments on the PNG benchmark show that our method achieves new state-of-the-art performance.<|reference_end|>
@article{li2024dynamic,
    title={Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic
  Narrative Grounding},
    author={Hongyu Li, Tianrui Hui, Zihan Ding, Jing Zhang, Bin Ma, Xiaoming Wei,
  Jizhong Han, Si Liu},
    journal={arXiv preprint arXiv:2409.08251},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.08251},
    primaryClass={cs.CV}
}