title: Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for
    Cartoon Captioning
summary: We present a novel multimodal preference dataset for creative tasks,
  consisting of over 250 million human ratings on more than 2.2 million captions,
  collected through crowdsourcing rating data for The New Yorker's weekly cartoon
  caption contest over the past eight years. This unique dataset supports the
  development and evaluation of multimodal large language models and
  preference-based fine-tuning algorithms for humorous caption generation. We
  propose novel benchmarks for judging the quality of model-generated captions,
  utilizing both GPT4 and human judgments to establish ranking-based evaluation
  strategies. Our experimental results highlight the limitations of current
  fine-tuning methods, such as RLHF and DPO, when applied to creative tasks.
  Furthermore, we demonstrate that even state-of-the-art models like GPT4 and
  Claude currently underperform top human contestants in generating humorous
  captions. As we conclude this extensive data collection effort, we release the
  entire preference dataset to the research community, fostering further
  advancements in AI humor generation and evaluation.
paper_authors._id: 6671b1468b5e22ce4f2ca4ad
paper_authors.name: Jifan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T21:26:52.590Z
paper_authors._id: 662a3a061bbfbf02afb1db90
paper_authors.avatarUrl: /avatars/f2cdeddb2810a3d9792b8c69c9e48ade.svg
paper_authors.fullname: Jifan Zhang
paper_authors.type: user
paper_authors.user: jifanz
paper_authors._id: 6671b1468b5e22ce4f2ca4ae
paper_authors.name: Lalit Jain
paper_authors._id: 6671b1468b5e22ce4f2ca4af
paper_authors.name: Yang Guo
paper_authors._id: 6671b1468b5e22ce4f2ca4b0
paper_authors.name: Jiayi Chen
paper_authors._id: 6671b1468b5e22ce4f2ca4b1
paper_authors.name: Kuan Lok Zhou
paper_authors._id: 6671b1468b5e22ce4f2ca4b2
paper_authors.name: Siddharth Suresh
paper_authors._id: 6671b1468b5e22ce4f2ca4b3
paper_authors.name: Andrew Wagenmaker
paper_authors._id: 6671b1468b5e22ce4f2ca4b4
paper_authors.name: Scott Sievert
paper_authors._id: 6671b1468b5e22ce4f2ca4b5
paper_authors.name: Timothy Rogers
paper_authors._id: 6671b1468b5e22ce4f2ca4b6
paper_authors.name: Kevin Jamieson
paper_authors._id: 6671b1468b5e22ce4f2ca4b7
paper_authors.name: Robert Mankoff
paper_authors._id: 6671b1468b5e22ce4f2ca4b8
paper_authors.name: Robert Nowak
paper_title: Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for
    Cartoon Captioning
paper_summary: We present a novel multimodal preference dataset for creative tasks,
  consisting of over 250 million human ratings on more than 2.2 million captions,
  collected through crowdsourcing rating data for The New Yorker's weekly cartoon
  caption contest over the past eight years. This unique dataset supports the
  development and evaluation of multimodal large language models and
  preference-based fine-tuning algorithms for humorous caption generation. We
  propose novel benchmarks for judging the quality of model-generated captions,
  utilizing both GPT4 and human judgments to establish ranking-based evaluation
  strategies. Our experimental results highlight the limitations of current
  fine-tuning methods, such as RLHF and DPO, when applied to creative tasks.
  Furthermore, we demonstrate that even state-of-the-art models like GPT4 and
  Claude currently underperform top human contestants in generating humorous
  captions. As we conclude this extensive data collection effort, we release the
  entire preference dataset to the research community, fostering further
  advancements in AI humor generation and evaluation.
paper_ai_keywords: multimodal large language models crowdsourcing rating data neural centrality measure-based fine-tuning algorithms GPT4 Claude RLHF DPO preference dataset AI humor generations and evaluation