Towards Generative Class Prompt Learning for Fine-grained Visual Recognition
<|reference_start|>Towards Generative Class Prompt Learning for Fine-grained Visual Recognition: Although foundational vision-language models (VLMs) have proven to be very successful for various semantic discrimination tasks, they still struggle to perform faithfully for fine-grained categorization. Moreover, foundational models trained on one domain do not generalize well on a different domain without fine-tuning. We attribute these to the limitations of the VLM's semantic representations and attempt to improve their fine-grained visual awareness using generative modeling. Specifically, we propose two novel methods: Generative Class Prompt Learning (GCPL) and Contrastive Multi-class Prompt Learning (CoMPLe). Utilizing text-to-image diffusion models, GCPL significantly improves the visio-linguistic synergy in class embeddings by conditioning on few-shot exemplars with learnable class prompts. CoMPLe builds on this foundation by introducing a contrastive learning component that encourages inter-class separation during the generative optimization process. Our empirical results demonstrate that such a generative class prompt learning approach substantially outperform existing methods, offering a better alternative to few shot image recognition challenges. The source code will be made available at: https://github.com/soumitri2001/GCPL.<|reference_end|>
@article{chattopadhyay2024towards,
    title={Towards Generative Class Prompt Learning for Fine-grained Visual
  Recognition},
    author={Soumitri Chattopadhyay, Sanket Biswas, Emanuele Vivoli, Josep Llad'os},
    journal={arXiv preprint arXiv:2409.01835},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.01835},
    primaryClass={cs.CV cs.CL}
}