title: MLLM-as-a-Judge for Image Safety without Human Labeling
summary: Image content safety has become a significant challenge with the rise of
  visual media on online platforms. Meanwhile, in the age of AI-generated content
  (AIGC), many image generation models are capable of producing harmful content,
  such as images containing sexual or violent material. Thus, it becomes crucial
  to identify such unsafe images based on established safety rules. Pre-trained
  Multimodal Large Language Models (MLLMs) offer potential in this regard, given
  their strong pattern recognition abilities. Existing approaches typically
  fine-tune MLLMs with human-labeled datasets, which however brings a series of
  drawbacks. First, relying on human annotators to label data following intricate
  and detailed guidelines is both expensive and labor-intensive. Furthermore,
  users of safety judgment systems may need to frequently update safety rules,
  making fine-tuning on human-based annotation more challenging. This raises the
  research question: Can we detect unsafe images by querying MLLMs in a zero-shot
  setting using a predefined safety constitution (a set of safety rules)? Our
  research showed that simply querying pre-trained MLLMs does not yield
  satisfactory results. This lack of effectiveness stems from factors such as the
  subjectivity of safety rules, the complexity of lengthy constitutions, and the
  inherent biases in the models. To address these challenges, we propose a
  MLLM-based method includes objectifying safety rules, assessing the relevance
  between rules and images, making quick judgments based on debiased token
  probabilities with logically complete yet simplified precondition chains for
  safety rules, and conducting more in-depth reasoning with cascaded
  chain-of-thought processes if necessary. Experiment results demonstrate that
  our method is highly effective for zero-shot image safety judgment tasks.
paper_authors._id: 6777591138f9a731d4f575bb
paper_authors.name: Zhenting Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:03:15.648Z
paper_authors._id: 64dfcc62e8b6f3f3baa950e0
paper_authors.avatarUrl: /avatars/21bbff67d46c08044efe2406575aa77e.svg
paper_authors.fullname: Zhenting Wang
paper_authors.type: user
paper_authors.user: ztwang
paper_authors._id: 6777591138f9a731d4f575bc
paper_authors.name: Shuming Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:53:03.995Z
paper_authors._id: 6563c6846ef2a1d0f989f11b
paper_authors.avatarUrl: /avatars/b03f6680ac3e718a1c38fec59a211b62.svg
paper_authors.fullname: Shuming Hu
paper_authors.type: user
paper_authors.user: shumingh
paper_authors._id: 6777591138f9a731d4f575bd
paper_authors.name: Shiyu Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:53:11.798Z
paper_authors._id: 667caf5baeba4a9f63860cf3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/667caf5baeba4a9f63860cf3/p4D4IXN1JgHwI8RGe5p6F.jpeg
paper_authors.fullname: Shiyu Zhao
paper_authors.type: user
paper_authors.user: xiaofeng-94
paper_authors._id: 6777591138f9a731d4f575be
paper_authors.name: Xiaowen Lin
paper_authors._id: 6777591138f9a731d4f575bf
paper_authors.name: Felix Juefei-Xu
paper_authors._id: 6777591138f9a731d4f575c0
paper_authors.name: Zhuowei Li
paper_authors._id: 6777591138f9a731d4f575c1
paper_authors.name: Ligong Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:54:08.096Z
paper_authors._id: 631e56ecf318ed8dfd3738cc
paper_authors.avatarUrl: /avatars/df4f71ac795cb8c251db7f30529dc2f8.svg
paper_authors.fullname: Ligong Han
paper_authors.type: user
paper_authors.user: ligongh
paper_authors._id: 6777591138f9a731d4f575c2
paper_authors.name: Harihar Subramanyam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:54:14.540Z
paper_authors._id: 6303da377373aacccd868d0f
paper_authors.avatarUrl: /avatars/88813105f863a0b2125fbe8bfffdec0e.svg
paper_authors.fullname: Harihar Subramanyam
paper_authors.type: user
paper_authors.user: Harihar
paper_authors._id: 6777591138f9a731d4f575c3
paper_authors.name: Li Chen
paper_authors._id: 6777591138f9a731d4f575c4
paper_authors.name: Jianfa Chen
paper_authors._id: 6777591138f9a731d4f575c5
paper_authors.name: Nan Jiang
paper_authors._id: 6777591138f9a731d4f575c6
paper_authors.name: Lingjuan Lyu
paper_authors._id: 6777591138f9a731d4f575c7
paper_authors.name: Shiqing Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:56:30.168Z
paper_authors._id: 644f91f317b6189cda55deea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/gX6s1BFk_JRKq4_HSQIcU.png
paper_authors.fullname: Shiqing Ma
paper_authors.type: user
paper_authors.user: Kuingsmile
paper_authors._id: 6777591138f9a731d4f575c8
paper_authors.name: Dimitris N. Metaxas
paper_authors._id: 6777591138f9a731d4f575c9
paper_authors.name: Ankit Jain
paper_title: MLLM-as-a-Judge for Image Safety without Human Labeling
paper_summary: Image content safety has become a significant challenge with the rise of
  visual media on online platforms. Meanwhile, in the age of AI-generated content
  (AIGC), many image generation models are capable of producing harmful content,
  such as images containing sexual or violent material. Thus, it becomes crucial
  to identify such unsafe images based on established safety rules. Pre-trained
  Multimodal Large Language Models (MLLMs) offer potential in this regard, given
  their strong pattern recognition abilities. Existing approaches typically
  fine-tune MLLMs with human-labeled datasets, which however brings a series of
  drawbacks. First, relying on human annotators to label data following intricate
  and detailed guidelines is both expensive and labor-intensive. Furthermore,
  users of safety judgment systems may need to frequently update safety rules,
  making fine-tuning on human-based annotation more challenging. This raises the
  research question: Can we detect unsafe images by querying MLLMs in a zero-shot
  setting using a predefined safety constitution (a set of safety rules)? Our
  research showed that simply querying pre-trained MLLMs does not yield
  satisfactory results. This lack of effectiveness stems from factors such as the
  subjectivity of safety rules, the complexity of lengthy constitutions, and the
  inherent biases in the models. To address these challenges, we propose a
  MLLM-based method includes objectifying safety rules, assessing the relevance
  between rules and images, making quick judgments based on debiased token
  probabilities with logically complete yet simplified precondition chains for
  safety rules, and conducting more in-depth reasoning with cascaded
  chain-of-thought processes if necessary. Experiment results demonstrate that
  our method is highly effective for zero-shot image safety judgment tasks.
paper_ai_summary: A method using Multimodal Large Language Models for zero-shot detection of unsafe images improves upon human-labeled fine-tuning by objectifying safety rules and using cascaded reasoning for complex judgments.
paper_ai_keywords: Multimodal Large Language Models zero-shot setting safety rules token probabilities precondition chains cascaded chain-of-thought processes