title: More Context, Less Distraction: Visual Classification by Inferring and
    Conditioning on Contextual Attributes
summary: CLIP, as a foundational vision language model, is widely used in zero-shot
  image classification due to its ability to understand various visual concepts
  and natural language descriptions. However, how to fully leverage CLIP's
  unprecedented human-like understanding capabilities to achieve better zero-shot
  classification is still an open question. This paper draws inspiration from the
  human visual perception process: a modern neuroscience view suggests that in
  classifying an object, humans first infer its class-independent attributes
  (e.g., background and orientation) which help separate the foreground object
  from the background, and then make decisions based on this information.
  Inspired by this, we observe that providing CLIP with contextual attributes
  improves zero-shot classification and mitigates reliance on spurious features.
  We also observe that CLIP itself can reasonably infer the attributes from an
  image. With these observations, we propose a training-free, two-step zero-shot
  classification method named PerceptionCLIP. Given an image, it first infers
  contextual attributes (e.g., background) and then performs object
  classification conditioning on them. Our experiments show that PerceptionCLIP
  achieves better generalization, group robustness, and better interpretability.
  For example, PerceptionCLIP with ViT-L/14 improves the worst group accuracy by
  16.5% on the Waterbirds dataset and by 3.5% on CelebA.
paper_authors._id: 64cb08617a4f236357164def
paper_authors.name: Bang An
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-08-03T02:30:29.418Z
paper_authors._id: 645071555fdcff143ad8e07f
paper_authors.avatarUrl: /avatars/a59f4e518a500e1c58b3a6d688644432.svg
paper_authors.fullname: Bang An
paper_authors.type: user
paper_authors.user: bangan
paper_authors._id: 64cb08617a4f236357164df0
paper_authors.name: Sicheng Zhu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-08-03T03:18:58.400Z
paper_authors._id: 64cac1fd8174e45ae03f12fa
paper_authors.avatarUrl: /avatars/21a80c732160a9a594215cd7f012f619.svg
paper_authors.fullname: Sicheng Zhu
paper_authors.type: user
paper_authors.user: sichengzhu
paper_authors._id: 64cb08617a4f236357164df1
paper_authors.name: Michael-Andrei Panaitescu-Liess
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-01-21T05:26:18.694Z
paper_authors._id: 65acaaa3b68db4f26e9e71d5
paper_authors.avatarUrl: /avatars/41da4b51d212d84c2b0eb8fca2eea477.svg
paper_authors.fullname: Michael-Andrei Panaitescu-Liess
paper_authors.type: user
paper_authors.user: michaelpanaitescu
paper_authors._id: 64cb08617a4f236357164df2
paper_authors.name: Chaithanya Kumar Mummadi
paper_authors._id: 64cb08617a4f236357164df3
paper_authors.name: Furong Huang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-08-03T15:19:48.595Z
paper_authors._id: 64cbc3e2a257a3212c00a115
paper_authors.avatarUrl: /avatars/836e61be4aeda2080ddf2db9f2626cc6.svg
paper_authors.fullname: Furong Huang Lab at UMD
paper_authors.type: user
paper_authors.user: furongh-lab
paper_title: More Context, Less Distraction: Visual Classification by Inferring and
    Conditioning on Contextual Attributes
paper_summary: CLIP, as a foundational vision language model, is widely used in zero-shot
  image classification due to its ability to understand various visual concepts
  and natural language descriptions. However, how to fully leverage CLIP's
  unprecedented human-like understanding capabilities to achieve better zero-shot
  classification is still an open question. This paper draws inspiration from the
  human visual perception process: a modern neuroscience view suggests that in
  classifying an object, humans first infer its class-independent attributes
  (e.g., background and orientation) which help separate the foreground object
  from the background, and then make decisions based on this information.
  Inspired by this, we observe that providing CLIP with contextual attributes
  improves zero-shot classification and mitigates reliance on spurious features.
  We also observe that CLIP itself can reasonably infer the attributes from an
  image. With these observations, we propose a training-free, two-step zero-shot
  classification method named PerceptionCLIP. Given an image, it first infers
  contextual attributes (e.g., background) and then performs object
  classification conditioning on them. Our experiments show that PerceptionCLIP
  achieves better generalization, group robustness, and better interpretability.
  For example, PerceptionCLIP with ViT-L/14 improves the worst group accuracy by
  16.5% on the Waterbirds dataset and by 3.5% on CelebA.
paper_ai_keywords: CLIP vision language model zero-shot image classification visual concepts natural language descriptions human visual perception process human neuroscience view class-independent attributes foreground object background contextual attributes spurious features training-free two-step zero-shot classification PerceptionCLIP generalization group robustness interpretability Waterbirds dataset CelebA ViT-L/14