title: StructEval: Benchmarking LLMs' Capabilities to Generate Structural
    Outputs
summary: As Large Language Models (LLMs) become integral to software development
  workflows, their ability to generate structured outputs has become critically
  important. We introduce StructEval, a comprehensive benchmark for evaluating
  LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and
  renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks,
  StructEval systematically evaluates structural fidelity across diverse formats
  through two paradigms: 1) generation tasks, producing structured output from
  natural language prompts, and 2) conversion tasks, translating between
  structured formats. Our benchmark encompasses 18 formats and 44 types of task,
  with novel metrics for format adherence and structural correctness. Results
  reveal significant performance gaps, even state-of-the-art models like o1-mini
  achieve only 75.58 average score, with open-source alternatives lagging
  approximately 10 points behind. We find generation tasks more challenging than
  conversion tasks, and producing correct visual content more difficult than
  generating text-only structures.
paper_authors._id: 6835744884d4600675a4449c
paper_authors.name: Jialin Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:00:22.494Z
paper_authors._id: 65d78a07ab0790921b087b68
paper_authors.avatarUrl: /avatars/869bbdb12f12b0dd46ee9f503416288e.svg
paper_authors.fullname: Jialin Yang
paper_authors.type: user
paper_authors.user: Sherloc512
paper_authors._id: 6835744884d4600675a4449d
paper_authors.name: Dongfu Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:58:01.881Z
paper_authors._id: 62567c86d444a9b5a0ec51c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62567c86d444a9b5a0ec51c1/1vXJf2uGztPcXpkwyTBr6.png
paper_authors.fullname: Dongfu Jiang
paper_authors.type: user
paper_authors.user: DongfuJiang
paper_authors._id: 6835744884d4600675a4449e
paper_authors.name: Lipeng He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:00:20.417Z
paper_authors._id: 6433707307bad11484af1d2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6433707307bad11484af1d2a/w5zB-zstJzY561n6q7m4D.jpeg
paper_authors.fullname: Lipeng (Tony) He
paper_authors.type: user
paper_authors.user: ttttonyhe
paper_authors._id: 6835744884d4600675a4449f
paper_authors.name: Sherman Siu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:57:49.231Z
paper_authors._id: 630d73387dacb93b335b00e9
paper_authors.avatarUrl: /avatars/2ded3550aedf77a58175a66319d01820.svg
paper_authors.fullname: Sherman Siu
paper_authors.type: user
paper_authors.user: shermansiu
paper_authors._id: 6835744884d4600675a444a0
paper_authors.name: Yuxuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:59:36.788Z
paper_authors._id: 63b908d0e3c78740d8e950d0
paper_authors.avatarUrl: /avatars/3e80075e92aebdfea712f70b00d5ec7d.svg
paper_authors.fullname: Yuxuan Zhang
paper_authors.type: user
paper_authors.user: Reacherx
paper_authors._id: 6835744884d4600675a444a1
paper_authors.name: Disen Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:57:33.699Z
paper_authors._id: 66c6853c13962d19a8395499
paper_authors.avatarUrl: /avatars/3066df4244a064e49880c19fd83c9426.svg
paper_authors.fullname: disen-liao
paper_authors.type: user
paper_authors.user: Dliao
paper_authors._id: 6835744884d4600675a444a2
paper_authors.name: Zhuofeng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:57:27.171Z
paper_authors._id: 66349404f2c753240d02952a
paper_authors.avatarUrl: /avatars/4f207cf5807d9629b9f4f7d13875b840.svg
paper_authors.fullname: ZhuofengLi
paper_authors.type: user
paper_authors.user: ZhuofengLi
paper_authors._id: 6835744884d4600675a444a3
paper_authors.name: Huaye Zeng
paper_authors._id: 6835744884d4600675a444a4
paper_authors.name: Yiming Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:57:12.783Z
paper_authors._id: 6721451d41cc176331607843
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HQ6_0b2Y-X1ORS6tAz2cq.png
paper_authors.fullname: Yiming Jia
paper_authors.type: user
paper_authors.user: jymmmmm
paper_authors._id: 6835744884d4600675a444a5
paper_authors.name: Haozhe Wang
paper_authors._id: 6835744884d4600675a444a6
paper_authors.name: Benjamin Schneider
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:56:55.022Z
paper_authors._id: 6669ffeecf27eed0f2f2581e
paper_authors.avatarUrl: /avatars/6a1012cce15da12fa3bd6f9e0f87d100.svg
paper_authors.fullname: Benjamin Schneider
paper_authors.type: user
paper_authors.user: BenSchneider
paper_authors._id: 6835744884d4600675a444a7
paper_authors.name: Chi Ruan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:56:34.920Z
paper_authors._id: 673b578551d8d86ed0abcb7c
paper_authors.avatarUrl: /avatars/41dbd6103456569efb707601930fec7e.svg
paper_authors.fullname: chiruan
paper_authors.type: user
paper_authors.user: chiruan
paper_authors._id: 6835744884d4600675a444a8
paper_authors.name: Wentao Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T15:49:46.788Z
paper_authors._id: 65c387c807a1445dfe1e9452
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65c387c807a1445dfe1e9452/t0VnwQh2wRZ9W_UGTZ8zt.jpeg
paper_authors.fullname: Wentao Ma
paper_authors.type: user
paper_authors.user: tonymwt
paper_authors._id: 6835744884d4600675a444a9
paper_authors.name: Zhiheng Lyu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:56:27.752Z
paper_authors._id: 6114ea319ae90b69cb29fc92
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6114ea319ae90b69cb29fc92/uw2n4iiXGD5v5iUocmbxE.jpeg
paper_authors.fullname: Zhiheng Lyu
paper_authors.type: user
paper_authors.user: cogito233
paper_authors._id: 6835744884d4600675a444aa
paper_authors.name: Yifei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:59:38.669Z
paper_authors._id: 683623338f559a703165971d
paper_authors.avatarUrl: /avatars/73bae160a706f831110f235df41a2ced.svg
paper_authors.fullname: Yifei Wang
paper_authors.type: user
paper_authors.user: yifei-wang
paper_authors._id: 6835744884d4600675a444ab
paper_authors.name: Yi Lu
paper_authors._id: 6835744884d4600675a444ac
paper_authors.name: Quy Duc Do
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:55:41.859Z
paper_authors._id: 65fa5fa1180d06d37c845d51
paper_authors.avatarUrl: /avatars/a04eb59f0a3a3b5800ae443a8eaab785.svg
paper_authors.fullname: Quy Duc Do
paper_authors.type: user
paper_authors.user: qd2do
paper_authors._id: 6835744884d4600675a444ad
paper_authors.name: Ziyan Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:55:33.868Z
paper_authors._id: 64778fb8168cb428e00f69b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64778fb8168cb428e00f69b0/D_XYg74zHG9K3HUJj_gD4.jpeg
paper_authors.fullname: Ziyan Jiang
paper_authors.type: user
paper_authors.user: ziyjiang
paper_authors._id: 6835744884d4600675a444ae
paper_authors.name: Ping Nie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:55:11.300Z
paper_authors._id: 65358802a920f38780b3248a
paper_authors.avatarUrl: /avatars/9415510b598079973c2b0436ad12db9c.svg
paper_authors.fullname: Ping Nie
paper_authors.type: user
paper_authors.user: pingnieuk
paper_authors._id: 6835744884d4600675a444af
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:54:56.382Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: StructEval: Benchmarking LLMs' Capabilities to Generate Structural
    Outputs
paper_summary: As Large Language Models (LLMs) become integral to software development
  workflows, their ability to generate structured outputs has become critically
  important. We introduce StructEval, a comprehensive benchmark for evaluating
  LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and
  renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks,
  StructEval systematically evaluates structural fidelity across diverse formats
  through two paradigms: 1) generation tasks, producing structured output from
  natural language prompts, and 2) conversion tasks, translating between
  structured formats. Our benchmark encompasses 18 formats and 44 types of task,
  with novel metrics for format adherence and structural correctness. Results
  reveal significant performance gaps, even state-of-the-art models like o1-mini
  achieve only 75.58 average score, with open-source alternatives lagging
  approximately 10 points behind. We find generation tasks more challenging than
  conversion tasks, and producing correct visual content more difficult than
  generating text-only structures.
paper_ai_summary: StructEval benchmarks Large Language Models for generating and converting structured outputs, highlighting performance gaps and challenges in producing visual content.
paper_ai_keywords: Large Language Models (LLMs) StructEval structured outputs JSON YAML CSV HTML React SVG generation tasks conversion tasks format adherence structural correctness