title: Struc-Bench: Are Large Language Models Really Good at Generating Complex
    Structured Data?
summary: Despite the power of Large Language Models (LLMs) like GPT-4, they still
  struggle with tasks that require generating complex, structured outputs. In
  this study, we assess the capability of Current LLMs in generating complex
  structured data and propose a structure-aware fine-tuning approach as a
  solution to improve this ability. To perform a comprehensive evaluation, we
  propose Struc-Bench, include five representative LLMs (i.e., GPT-NeoX 20B,
  GPT-3.5, GPT-4, and Vicuna) and evaluate them on our carefully constructed
  datasets spanning raw text, HTML, and LaTeX tables. Based on our analysis of
  current model performance, we identify specific common formatting errors and
  areas of potential improvement. To address complex formatting requirements, we
  utilize FormatCoT (Chain-of-Thought) to generate format instructions from
  target outputs. Our experiments show that our structure-aware fine-tuning
  method, when applied to LLaMA-7B, significantly improves adherence to natural
  language constraints, outperforming other evaluated LLMs. Based on these
  results, we present an ability map of model capabilities from six dimensions
  (i.e., coverage, formatting, reasoning, comprehension, pragmatics, and
  hallucination). This map highlights the weaknesses of LLMs in handling complex
  structured outputs and suggests promising directions for future work. Our code
  and models can be found at https://github.com/gersteinlab/Struc-Bench.
paper_authors._id: 65090cabe64ee37323aef98e
paper_authors.name: Xiangru Tang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-07T11:18:12.620Z
paper_authors._id: 63357c608adfa81faf2ac180
paper_authors.avatarUrl: /avatars/ae0314c644f882251baf59b9134fd36f.svg
paper_authors.fullname: Xiangru Tang
paper_authors.type: user
paper_authors.user: RTT1
paper_authors._id: 65090cabe64ee37323aef98f
paper_authors.name: Yiming Zong
paper_authors._id: 65090cabe64ee37323aef990
paper_authors.name: Yilun Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:39:37.084Z
paper_authors._id: 62f662bcc58915315c4eccea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
paper_authors.fullname: Yilun
paper_authors.type: user
paper_authors.user: yilunzhao
paper_authors._id: 65090cabe64ee37323aef991
paper_authors.name: Arman Cohan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:39:45.163Z
paper_authors._id: 5f5ba21188f57f65f951f255
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
paper_authors.fullname: Arman Cohan
paper_authors.type: user
paper_authors.user: armanc
paper_authors._id: 65090cabe64ee37323aef992
paper_authors.name: Mark Gerstein
paper_title: Struc-Bench: Are Large Language Models Really Good at Generating Complex
    Structured Data?
paper_summary: Despite the power of Large Language Models (LLMs) like GPT-4, they still
  struggle with tasks that require generating complex, structured outputs. In
  this study, we assess the capability of Current LLMs in generating complex
  structured data and propose a structure-aware fine-tuning approach as a
  solution to improve this ability. To perform a comprehensive evaluation, we
  propose Struc-Bench, include five representative LLMs (i.e., GPT-NeoX 20B,
  GPT-3.5, GPT-4, and Vicuna) and evaluate them on our carefully constructed
  datasets spanning raw text, HTML, and LaTeX tables. Based on our analysis of
  current model performance, we identify specific common formatting errors and
  areas of potential improvement. To address complex formatting requirements, we
  utilize FormatCoT (Chain-of-Thought) to generate format instructions from
  target outputs. Our experiments show that our structure-aware fine-tuning
  method, when applied to LLaMA-7B, significantly improves adherence to natural
  language constraints, outperforming other evaluated LLMs. Based on these
  results, we present an ability map of model capabilities from six dimensions
  (i.e., coverage, formatting, reasoning, comprehension, pragmatics, and
  hallucination). This map highlights the weaknesses of LLMs in handling complex
  structured outputs and suggests promising directions for future work. Our code
  and models can be found at https://github.com/gersteinlab/Struc-Bench.
paper_ai_keywords: structure-aware fine-tuning Struc-Bench GPT-NeoX 20B GPT-3.5 GPT-4 Vicuna FormatCoT (Chain-of-Thought) LLaMA-7B ability map coverage formatting reasoning comprehension pragmatics hallucination