  The age of social media is rife with memes. Understanding and detecting
harmful memes pose a significant challenge due to their implicit meaning that
is not explicitly conveyed through the surface text and image. However,
existing harmful meme detection approaches only recognize superficial
harm-indicative signals in an end-to-end classification manner but ignore
in-depth cognition of the meme text and image. In this paper, we attempt to
detect harmful memes based on advanced reasoning over the interplay of
multimodal information in memes. Inspired by the success of Large Language
Models (LLMs) on complex reasoning, we first conduct abductive reasoning with
LLMs. Then we propose a novel generative framework to learn reasonable thoughts
from LLMs for better multimodal fusion and lightweight fine-tuning, which
consists of two training stages: 1) Distill multimodal reasoning knowledge from
LLMs; and 2) Fine-tune the generative framework to infer harmfulness. Extensive
experiments conducted on three meme datasets demonstrate that our proposed
approach achieves superior performance than state-of-the-art methods on the
harmful meme detection task.

Computation and Language