title: LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large
    Multimodal Models
summary: With the rapid development of AI-generated content, the future internet may
  be inundated with synthetic data, making the discrimination of authentic and
  credible multimodal data increasingly challenging. Synthetic data detection has
  thus garnered widespread attention, and the performance of large multimodal
  models (LMMs) in this task has attracted significant interest. LMMs can provide
  natural language explanations for their authenticity judgments, enhancing the
  explainability of synthetic content detection. Simultaneously, the task of
  distinguishing between real and synthetic data effectively tests the
  perception, knowledge, and reasoning capabilities of LMMs. In response, we
  introduce LOKI, a novel benchmark designed to evaluate the ability of LMMs to
  detect synthetic data across multiple modalities. LOKI encompasses video,
  image, 3D, text, and audio modalities, comprising 18K carefully curated
  questions across 26 subcategories with clear difficulty levels. The benchmark
  includes coarse-grained judgment and multiple-choice questions, as well as
  fine-grained anomaly selection and explanation tasks, allowing for a
  comprehensive analysis of LMMs. We evaluated 22 open-source LMMs and 6
  closed-source models on LOKI, highlighting their potential as synthetic data
  detectors and also revealing some limitations in the development of LMM
  capabilities. More information about LOKI can be found at
  https://opendatalab.github.io/LOKI/
paper_authors._id: 670dd110aa00e89d82f7011f
paper_authors.name: Junyan Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T09:28:39.560Z
paper_authors._id: 66978ee0b8656f6506b4acb2
paper_authors.avatarUrl: /avatars/298acb8222e189fce4368985ee5374a1.svg
paper_authors.fullname: Junyan Ye
paper_authors.type: user
paper_authors.user: Yejy53
paper_authors._id: 670dd110aa00e89d82f70120
paper_authors.name: Baichuan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:10:58.373Z
paper_authors._id: 646325085897b675c65aea0f
paper_authors.avatarUrl: /avatars/28ce7388f9318b49bdd0a5594c0f6732.svg
paper_authors.fullname: Baichuan Zhou
paper_authors.type: user
paper_authors.user: bczhou
paper_authors._id: 670dd110aa00e89d82f70121
paper_authors.name: Zilong Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:44:20.395Z
paper_authors._id: 6487e158f675b4a7867f45fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6487e158f675b4a7867f45fa/J0sls6zZ682o-SH7iQs7B.jpeg
paper_authors.fullname: Zilong Huang
paper_authors.type: user
paper_authors.user: SereinH
paper_authors._id: 670dd110aa00e89d82f70122
paper_authors.name: Junan Zhang
paper_authors._id: 670dd110aa00e89d82f70123
paper_authors.name: Tianyi Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:00.531Z
paper_authors._id: 641802c31f1f3b0fa812225c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641802c31f1f3b0fa812225c/i-OFWm8d37ofsmuvvAKYR.png
paper_authors.fullname: becca_bai
paper_authors.type: user
paper_authors.user: beccabai
paper_authors._id: 670dd110aa00e89d82f70124
paper_authors.name: Hengrui Kang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:24:34.153Z
paper_authors._id: 6617f09e4f9cea969619bfbd
paper_authors.avatarUrl: /avatars/4dbc9c7b6814edcdab334ed3f0a1a4dd.svg
paper_authors.fullname: Heng Rui Kang
paper_authors.type: user
paper_authors.user: ruikang2002
paper_authors._id: 670dd110aa00e89d82f70125
paper_authors.name: Jun He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T13:39:45.836Z
paper_authors._id: 670ddb69d6ac6394419d88c5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/XxnGNaX3FWug4aiZVjg93.png
paper_authors.fullname: Jun He
paper_authors.type: user
paper_authors.user: JunHe0915
paper_authors._id: 670dd110aa00e89d82f70126
paper_authors.name: Honglin Lin
paper_authors._id: 670dd110aa00e89d82f70127
paper_authors.name: Zihao Wang
paper_authors._id: 670dd110aa00e89d82f70128
paper_authors.name: Tong Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:21:45.971Z
paper_authors._id: 6310929fda0aaad2f70397a1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6310929fda0aaad2f70397a1/YoHeYZezmCOXwL-qsiZOc.png
paper_authors.fullname: Tong Wu
paper_authors.type: user
paper_authors.user: wutong16
paper_authors._id: 670dd110aa00e89d82f70129
paper_authors.name: Zhizheng Wu
paper_authors._id: 670dd110aa00e89d82f7012a
paper_authors.name: Yiping Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:22:50.336Z
paper_authors._id: 653a22c6a6e8c78bc13f9d3c
paper_authors.avatarUrl: /avatars/c9ca843aded54960c661f9eb232d6d60.svg
paper_authors.fullname: Yiping Chen
paper_authors.type: user
paper_authors.user: chenchenchenchenya
paper_authors._id: 670dd110aa00e89d82f7012b
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:22:28.798Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 670dd110aa00e89d82f7012c
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:22:43.987Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 670dd110aa00e89d82f7012d
paper_authors.name: Weijia Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T13:22:37.451Z
paper_authors._id: 66d5b56c77a026c3d2086a79
paper_authors.avatarUrl: /avatars/45da07fd82fd455955faa05b27a6393f.svg
paper_authors.fullname: Weijia Li
paper_authors.type: user
paper_authors.user: liweijia
paper_title: LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large
    Multimodal Models
paper_summary: With the rapid development of AI-generated content, the future internet may
  be inundated with synthetic data, making the discrimination of authentic and
  credible multimodal data increasingly challenging. Synthetic data detection has
  thus garnered widespread attention, and the performance of large multimodal
  models (LMMs) in this task has attracted significant interest. LMMs can provide
  natural language explanations for their authenticity judgments, enhancing the
  explainability of synthetic content detection. Simultaneously, the task of
  distinguishing between real and synthetic data effectively tests the
  perception, knowledge, and reasoning capabilities of LMMs. In response, we
  introduce LOKI, a novel benchmark designed to evaluate the ability of LMMs to
  detect synthetic data across multiple modalities. LOKI encompasses video,
  image, 3D, text, and audio modalities, comprising 18K carefully curated
  questions across 26 subcategories with clear difficulty levels. The benchmark
  includes coarse-grained judgment and multiple-choice questions, as well as
  fine-grained anomaly selection and explanation tasks, allowing for a
  comprehensive analysis of LMMs. We evaluated 22 open-source LMMs and 6
  closed-source models on LOKI, highlighting their potential as synthetic data
  detectors and also revealing some limitations in the development of LMM
  capabilities. More information about LOKI can be found at
  https://opendatalab.github.io/LOKI/
paper_ai_summary: LOKI is a benchmark for evaluating the synthetic data detection capabilities of large multimodal models across various modalities.
paper_ai_keywords: multimodal models synthetic data detection explainability natural language explanations video image 3D text audio benchmark coarse-grained judgment multiple-choice questions fine-grained anomaly selection explanation tasks