AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large
  Language Models on Harmfulness
The proliferation of multimodal memes in the social media era demands that
multimodal Large Language Models (mLLMs) effectively understand meme
harmfulness. Existing benchmarks for assessing mLLMs on harmful meme
understanding rely on accuracy-based, model-agnostic evaluations using static
datasets. These benchmarks are limited in their ability to provide up-to-date
and thorough assessments, as online memes evolve dynamically. To address this,
we propose AdamMeme, a flexible, agent-based evaluation framework that
adaptively probes the reasoning capabilities of mLLMs in deciphering meme
harmfulness. Through multi-agent collaboration, AdamMeme provides comprehensive
evaluations by iteratively updating the meme data with challenging samples,
thereby exposing specific limitations in how mLLMs interpret harmfulness.
Extensive experiments show that our framework systematically reveals the
varying performance of different target mLLMs, offering in-depth, fine-grained
analyses of model-specific weaknesses. Our code is available at
https://github.com/Lbotirx/AdamMeme.
686f7ef5d938c25d6844295f
Zixin Chen
686f7ef5d938c25d68442960
Hongzhan Lin
686f7ef5d938c25d68442961
Kaixin Li
686f7ef5d938c25d68442962
Ziyang Luo
686f7ef5d938c25d68442963
Zhen Ye
686f7ef5d938c25d68442964
Guang Chen
686f7ef5d938c25d68442965
Zhiyong Huang
686f7ef5d938c25d68442966
Jing Ma
AdamMeme: Adaptively Probe the Reasoning Capacity of Multimodal Large
  Language Models on Harmfulness
The proliferation of multimodal memes in the social media era demands that
multimodal Large Language Models (mLLMs) effectively understand meme
harmfulness. Existing benchmarks for assessing mLLMs on harmful meme
understanding rely on accuracy-based, model-agnostic evaluations using static
datasets. These benchmarks are limited in their ability to provide up-to-date
and thorough assessments, as online memes evolve dynamically. To address this,
we propose AdamMeme, a flexible, agent-based evaluation framework that
adaptively probes the reasoning capabilities of mLLMs in deciphering meme
harmfulness. Through multi-agent collaboration, AdamMeme provides comprehensive
evaluations by iteratively updating the meme data with challenging samples,
thereby exposing specific limitations in how mLLMs interpret harmfulness.
Extensive experiments show that our framework systematically reveals the
varying performance of different target mLLMs, offering in-depth, fine-grained
analyses of model-specific weaknesses. Our code is available at
https://github.com/Lbotirx/AdamMeme.
AdamMeme is an adaptive, agent-based evaluation framework that assesses multimodal Large Language Models' understanding of harmful memes by iteratively updating with challenging samples.
multimodal Large Language Models mLLMs meme harmfulness agent-based evaluation framework multi-agent collaboration meme data model-specific weaknesses