title: Interleaved Scene Graph for Interleaved Text-and-Image Generation
    Assessment
summary: Many real-world user queries (e.g. "How do to make egg fried rice?") could
  benefit from systems capable of generating responses with both textual steps
  with accompanying images, similar to a cookbook. Models designed to generate
  interleaved text and images face challenges in ensuring consistency within and
  across these modalities. To address these challenges, we present ISG, a
  comprehensive evaluation framework for interleaved text-and-image generation.
  ISG leverages a scene graph structure to capture relationships between text and
  image blocks, evaluating responses on four levels of granularity: holistic,
  structural, block-level, and image-specific. This multi-tiered evaluation
  allows for a nuanced assessment of consistency, coherence, and accuracy, and
  provides interpretable question-answer feedback. In conjunction with ISG, we
  introduce a benchmark, ISG-Bench, encompassing 1,150 samples across 8
  categories and 21 subcategories. This benchmark dataset includes complex
  language-vision dependencies and golden answers to evaluate models effectively
  on vision-centric tasks such as style transfer, a challenging area for current
  models. Using ISG-Bench, we demonstrate that recent unified vision-language
  models perform poorly on generating interleaved content. While compositional
  approaches that combine separate language and image models show a 111%
  improvement over unified models at the holistic level, their performance
  remains suboptimal at both block and image levels. To facilitate future work,
  we develop ISG-Agent, a baseline agent employing a "plan-execute-refine"
  pipeline to invoke tools, achieving a 122% performance improvement.
paper_authors._id: 67467ee679406f42a1466a07
paper_authors.name: Dongping Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-27T22:20:26.369Z
paper_authors._id: 643be8879f5d314db2d9ed23
paper_authors.avatarUrl: /avatars/64e9bb2c4e10fbe03e2b81afedf40865.svg
paper_authors.fullname: Chen Dongping
paper_authors.type: user
paper_authors.user: shuaishuaicdp
paper_authors._id: 67467ee679406f42a1466a08
paper_authors.name: Ruoxi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T10:39:29.970Z
paper_authors._id: 65a135df18b7d7c225737173
paper_authors.avatarUrl: /avatars/a186f776011bb62c54fd73e787ff8415.svg
paper_authors.fullname: Ruoxi Chen
paper_authors.type: user
paper_authors.user: Dipsy0830
paper_authors._id: 67467ee679406f42a1466a09
paper_authors.name: Shu Pu
paper_authors._id: 67467ee679406f42a1466a0a
paper_authors.name: Zhaoyi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T10:39:43.179Z
paper_authors._id: 6695ca27bcd81f395ec16e6d
paper_authors.avatarUrl: /avatars/6f70cd2fe6d8316dfcf555b3d8cf706c.svg
paper_authors.fullname: Zhaoyi LIU
paper_authors.type: user
paper_authors.user: lzy8465
paper_authors._id: 67467ee679406f42a1466a0b
paper_authors.name: Yanru Wu
paper_authors._id: 67467ee679406f42a1466a0c
paper_authors.name: Caixi Chen
paper_authors._id: 67467ee679406f42a1466a0d
paper_authors.name: Benlin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T10:39:59.992Z
paper_authors._id: 6326723c21c98f83d6ad128e
paper_authors.avatarUrl: /avatars/6c208ff815c037d0a6a2ada3198e7fd4.svg
paper_authors.fullname: Benlin Liu
paper_authors.type: user
paper_authors.user: Tim666
paper_authors._id: 67467ee679406f42a1466a0e
paper_authors.name: Yue Huang
paper_authors._id: 67467ee679406f42a1466a0f
paper_authors.name: Yao Wan
paper_authors._id: 67467ee679406f42a1466a10
paper_authors.name: Pan Zhou
paper_authors._id: 67467ee679406f42a1466a11
paper_authors.name: Ranjay Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T10:39:19.626Z
paper_authors._id: 66429868ab89e3a3a85668b0
paper_authors.avatarUrl: /avatars/170e0daa454838deee2bf946f7118651.svg
paper_authors.fullname: Ranjay Krishna
paper_authors.type: user
paper_authors.user: ranjaykrishna
paper_title: Interleaved Scene Graph for Interleaved Text-and-Image Generation
    Assessment
paper_summary: Many real-world user queries (e.g. "How do to make egg fried rice?") could
  benefit from systems capable of generating responses with both textual steps
  with accompanying images, similar to a cookbook. Models designed to generate
  interleaved text and images face challenges in ensuring consistency within and
  across these modalities. To address these challenges, we present ISG, a
  comprehensive evaluation framework for interleaved text-and-image generation.
  ISG leverages a scene graph structure to capture relationships between text and
  image blocks, evaluating responses on four levels of granularity: holistic,
  structural, block-level, and image-specific. This multi-tiered evaluation
  allows for a nuanced assessment of consistency, coherence, and accuracy, and
  provides interpretable question-answer feedback. In conjunction with ISG, we
  introduce a benchmark, ISG-Bench, encompassing 1,150 samples across 8
  categories and 21 subcategories. This benchmark dataset includes complex
  language-vision dependencies and golden answers to evaluate models effectively
  on vision-centric tasks such as style transfer, a challenging area for current
  models. Using ISG-Bench, we demonstrate that recent unified vision-language
  models perform poorly on generating interleaved content. While compositional
  approaches that combine separate language and image models show a 111%
  improvement over unified models at the holistic level, their performance
  remains suboptimal at both block and image levels. To facilitate future work,
  we develop ISG-Agent, a baseline agent employing a "plan-execute-refine"
  pipeline to invoke tools, achieving a 122% performance improvement.
paper_ai_keywords: scene graph structure interleaved text-and-image generation holistic evaluation structural evaluation block-level evaluation image-specific evaluation multi-tiered evaluation consistency assessment coherence assessment accuracy assessment ISG-Agent plan-execute-refine pipeline unified vision-language models compositional approaches vision-centric tasks style transfer