title: Inject Semantic Concepts into Image Tagging for Open-Set Recognition
summary: In this paper, we introduce the Recognize Anything Plus Model~(RAM++), a
  fundamental image recognition model with strong open-set recognition
  capabilities, by injecting semantic concepts into image tagging training
  framework. Previous approaches are either image tagging models constrained by
  limited semantics, or vision-language models with shallow interaction for
  suboptimal performance in multi-tag recognition. In contrast, RAM++ integrates
  image-text alignment and image-tagging within a unified fine-grained
  interaction framework based on image-tags-text triplets. This design enables
  RAM++ not only excel in identifying predefined categories, but also
  significantly augment the recognition ability in open-set categories. Moreover,
  RAM++ employs large language models~(LLMs) to generate diverse visual tag
  descriptions, pioneering the integration of LLM's knowledge into image tagging
  training. This approach empowers RAM++ to integrate visual description concepts
  for open-set recognition during inference. Evaluations on comprehensive image
  recognition benchmarks demonstrate RAM++ exceeds existing state-of-the-art
  (SOTA) fundamental image recognition models on most aspects. Specifically, for
  predefined common-used tag categories, RAM++ showcases 10.2 mAP and 15.4 mAP
  enhancements over CLIP on OpenImages and ImageNet. For open-set categories
  beyond predefined, RAM++ records improvements of 5 mAP and 6.4 mAP over CLIP
  and RAM respectively on OpenImages. For diverse human-object interaction
  phrases, RAM++ achieves 7.8 mAP and 4.7 mAP improvements on the HICO benchmark.
  Code, datasets and pre-trained models are available at
  https://github.com/xinyu1205/recognize-anything.
paper_authors._id: 6538858bbf790526ef3ba93f
paper_authors.name: Xinyu Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:50:59.370Z
paper_authors._id: 63bf7ba8da08ed0544ff20e9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1673493776367-63bf7ba8da08ed0544ff20e9.jpeg
paper_authors.fullname: Xinyu Huang
paper_authors.type: user
paper_authors.user: xinyu1205
paper_authors._id: 6538858bbf790526ef3ba940
paper_authors.name: Yi-Jie Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:51:18.420Z
paper_authors._id: 648136af7d65d9ac17346bac
paper_authors.avatarUrl: /avatars/7fe65fe712b69f16d81123f0d67719ab.svg
paper_authors.fullname: Yijie Huang
paper_authors.type: user
paper_authors.user: asdfagafgs
paper_authors._id: 6538858bbf790526ef3ba941
paper_authors.name: Youcai Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:51:26.531Z
paper_authors._id: 6480327e136d429d0671ecd6
paper_authors.avatarUrl: /avatars/306decb0cee314df2c4bf8ad61938147.svg
paper_authors.fullname: Youcai Zhang
paper_authors.type: user
paper_authors.user: Coler
paper_authors._id: 6538858bbf790526ef3ba942
paper_authors.name: Weiwei Tian
paper_authors._id: 6538858bbf790526ef3ba943
paper_authors.name: Rui Feng
paper_authors._id: 6538858bbf790526ef3ba944
paper_authors.name: Yuejie Zhang
paper_authors._id: 6538858bbf790526ef3ba945
paper_authors.name: Yanchun Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:52:12.964Z
paper_authors._id: 62a69c16db0b844f8b15e6e1
paper_authors.avatarUrl: /avatars/971811e630bd5354a008f6abe3bffcce.svg
paper_authors.fullname: yanchun xie
paper_authors.type: user
paper_authors.user: xyc
paper_authors._id: 6538858bbf790526ef3ba946
paper_authors.name: Yaqian Li
paper_authors._id: 6538858bbf790526ef3ba947
paper_authors.name: Lei Zhang
paper_title: Inject Semantic Concepts into Image Tagging for Open-Set Recognition
paper_summary: In this paper, we introduce the Recognize Anything Plus Model~(RAM++), a
  fundamental image recognition model with strong open-set recognition
  capabilities, by injecting semantic concepts into image tagging training
  framework. Previous approaches are either image tagging models constrained by
  limited semantics, or vision-language models with shallow interaction for
  suboptimal performance in multi-tag recognition. In contrast, RAM++ integrates
  image-text alignment and image-tagging within a unified fine-grained
  interaction framework based on image-tags-text triplets. This design enables
  RAM++ not only excel in identifying predefined categories, but also
  significantly augment the recognition ability in open-set categories. Moreover,
  RAM++ employs large language models~(LLMs) to generate diverse visual tag
  descriptions, pioneering the integration of LLM's knowledge into image tagging
  training. This approach empowers RAM++ to integrate visual description concepts
  for open-set recognition during inference. Evaluations on comprehensive image
  recognition benchmarks demonstrate RAM++ exceeds existing state-of-the-art
  (SOTA) fundamental image recognition models on most aspects. Specifically, for
  predefined common-used tag categories, RAM++ showcases 10.2 mAP and 15.4 mAP
  enhancements over CLIP on OpenImages and ImageNet. For open-set categories
  beyond predefined, RAM++ records improvements of 5 mAP and 6.4 mAP over CLIP
  and RAM respectively on OpenImages. For diverse human-object interaction
  phrases, RAM++ achieves 7.8 mAP and 4.7 mAP improvements on the HICO benchmark.
  Code, datasets and pre-trained models are available at
  https://github.com/xinyu1205/recognize-anything.
paper_ai_summary: RAM++, a unified image-text interaction framework, enhances open-set recognition and predefined category identification by integrating large language models and semantic concepts into image tagging.
paper_ai_keywords: RAM++ open-set recognition image tagging image-text alignment image-tags-text triplets large language models visual description concepts OpenImages ImageNet HICO benchmark mAP