title: Can Large Language Models Understand Symbolic Graphics Programs?
summary: Assessing the capabilities of large language models (LLMs) is often
  challenging, in part, because it is hard to find tasks to which they have not
  been exposed during training. We take one step to address this challenge by
  turning to a new task: focusing on symbolic graphics programs, which are a
  popular representation for graphics content that procedurally generates visual
  data. LLMs have shown exciting promise towards program synthesis, but do they
  understand symbolic graphics programs? Unlike conventional programs, symbolic
  graphics programs can be translated to graphics content. Here, we characterize
  an LLM's understanding of symbolic programs in terms of their ability to answer
  questions related to the graphics content. This task is challenging as the
  questions are difficult to answer from the symbolic programs alone -- yet, they
  would be easy to answer from the corresponding graphics content as we verify
  through a human experiment. To understand symbolic programs, LLMs may need to
  possess the ability to imagine how the corresponding graphics content would
  look without directly accessing the rendered visual content. We use this task
  to evaluate LLMs by creating a large benchmark for the semantic understanding
  of symbolic graphics programs. This benchmark is built via program-graphics
  correspondence, hence requiring minimal human efforts. We evaluate current LLMs
  on our benchmark to elucidate a preliminary assessment of their ability to
  reason about visual scenes from programs. We find that this task distinguishes
  existing LLMs and models considered good at reasoning perform better. Lastly,
  we introduce Symbolic Instruction Tuning (SIT) to improve this ability.
  Specifically, we query GPT4-o with questions and images generated by symbolic
  programs. Such data are then used to finetune an LLM. We also find that SIT
  data can improve the general instruction following ability of LLMs.
paper_authors._id: 66beb9de6e746f4f3ca27cb7
paper_authors.name: Zeju Qiu
paper_authors._id: 66beb9de6e746f4f3ca27cb8
paper_authors.name: Weiyang Liu
paper_authors._id: 66beb9de6e746f4f3ca27cb9
paper_authors.name: Haiwen Feng
paper_authors._id: 66beb9de6e746f4f3ca27cba
paper_authors.name: Zhen Liu
paper_authors._id: 66beb9de6e746f4f3ca27cbb
paper_authors.name: Tim Z. Xiao
paper_authors._id: 66beb9de6e746f4f3ca27cbc
paper_authors.name: Katherine M. Collins
paper_authors._id: 66beb9de6e746f4f3ca27cbd
paper_authors.name: Joshua B. Tenenbaum
paper_authors._id: 66beb9de6e746f4f3ca27cbe
paper_authors.name: Adrian Weller
paper_authors._id: 66beb9de6e746f4f3ca27cbf
paper_authors.name: Michael J. Black
paper_authors._id: 66beb9de6e746f4f3ca27cc0
paper_authors.name: Bernhard Schölkopf
paper_title: Can Large Language Models Understand Symbolic Graphics Programs?
paper_summary: Assessing the capabilities of large language models (LLMs) is often
  challenging, in part, because it is hard to find tasks to which they have not
  been exposed during training. We take one step to address this challenge by
  turning to a new task: focusing on symbolic graphics programs, which are a
  popular representation for graphics content that procedurally generates visual
  data. LLMs have shown exciting promise towards program synthesis, but do they
  understand symbolic graphics programs? Unlike conventional programs, symbolic
  graphics programs can be translated to graphics content. Here, we characterize
  an LLM's understanding of symbolic programs in terms of their ability to answer
  questions related to the graphics content. This task is challenging as the
  questions are difficult to answer from the symbolic programs alone -- yet, they
  would be easy to answer from the corresponding graphics content as we verify
  through a human experiment. To understand symbolic programs, LLMs may need to
  possess the ability to imagine how the corresponding graphics content would
  look without directly accessing the rendered visual content. We use this task
  to evaluate LLMs by creating a large benchmark for the semantic understanding
  of symbolic graphics programs. This benchmark is built via program-graphics
  correspondence, hence requiring minimal human efforts. We evaluate current LLMs
  on our benchmark to elucidate a preliminary assessment of their ability to
  reason about visual scenes from programs. We find that this task distinguishes
  existing LLMs and models considered good at reasoning perform better. Lastly,
  we introduce Symbolic Instruction Tuning (SIT) to improve this ability.
  Specifically, we query GPT4-o with questions and images generated by symbolic
  programs. Such data are then used to finetune an LLM. We also find that SIT
  data can improve the general instruction following ability of LLMs.
paper_ai_summary: A benchmark evaluates large language models' understanding of symbolic graphics programs by assessing their ability to answer questions about the generated graphics content.
paper_ai_keywords: large language models (LLMs) symbolic graphics programs program synthesis semantic understanding benchmark program-graphics correspondence Symbolic Instruction Tuning (SIT) finetune instruction following