title: BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing
summary: 3D graphics editing is crucial in applications like movie production and game
  design, yet it remains a time-consuming process that demands highly specialized
  domain expertise. Automating this process is challenging because graphical
  editing requires performing a variety of tasks, each requiring distinct skill
  sets. Recently, vision-language models (VLMs) have emerged as a powerful
  framework for automating the editing process, but their development and
  evaluation are bottlenecked by the lack of a comprehensive benchmark that
  requires human-level perception and presents real-world editing complexity. In
  this work, we present BlenderGym, the first comprehensive VLM system benchmark
  for 3D graphics editing. BlenderGym evaluates VLM systems through code-based 3D
  reconstruction tasks. We evaluate closed- and open-source VLM systems and
  observe that even the state-of-the-art VLM system struggles with tasks
  relatively easy for human Blender users. Enabled by BlenderGym, we study how
  inference scaling techniques impact VLM's performance on graphics editing
  tasks. Notably, our findings reveal that the verifier used to guide the scaling
  of generation can itself be improved through inference scaling, complementing
  recent insights on inference scaling of LLM generation in coding and math
  tasks. We further show that inference compute is not uniformly effective and
  can be optimized by strategically distributing it between generation and
  verification.
paper_authors._id: 67fd668da1510fb6779d80af
paper_authors.name: Yunqi Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:55:27.029Z
paper_authors._id: 649c8613667e8e1426b80c49
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/WcSUg5ytefyeoKIAdabM2.png
paper_authors.fullname: Yunqi Gu(Richard)
paper_authors.type: user
paper_authors.user: richard-guyunqi
paper_authors._id: 67fd668da1510fb6779d80b0
paper_authors.name: Ian Huang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-14T19:48:33.297Z
paper_authors._id: 639ce51c66106be14370f510
paper_authors.avatarUrl: /avatars/74f68f5951fff9c02c7607ff197812aa.svg
paper_authors.fullname: Ian Huang
paper_authors.type: user
paper_authors.user: ianhuang
paper_authors._id: 67fd668da1510fb6779d80b1
paper_authors.name: Jihyeon Je
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-14T20:04:18.253Z
paper_authors._id: 65b96839f8f4f76f9100381b
paper_authors.avatarUrl: /avatars/8eb3017043b86468ab324c6795382055.svg
paper_authors.fullname: Jihyeon Je
paper_authors.type: user
paper_authors.user: jihyeonj
paper_authors._id: 67fd668da1510fb6779d80b2
paper_authors.name: Guandao Yang
paper_authors._id: 67fd668da1510fb6779d80b3
paper_authors.name: Leonidas Guibas
paper_title: BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing
paper_summary: 3D graphics editing is crucial in applications like movie production and game
  design, yet it remains a time-consuming process that demands highly specialized
  domain expertise. Automating this process is challenging because graphical
  editing requires performing a variety of tasks, each requiring distinct skill
  sets. Recently, vision-language models (VLMs) have emerged as a powerful
  framework for automating the editing process, but their development and
  evaluation are bottlenecked by the lack of a comprehensive benchmark that
  requires human-level perception and presents real-world editing complexity. In
  this work, we present BlenderGym, the first comprehensive VLM system benchmark
  for 3D graphics editing. BlenderGym evaluates VLM systems through code-based 3D
  reconstruction tasks. We evaluate closed- and open-source VLM systems and
  observe that even the state-of-the-art VLM system struggles with tasks
  relatively easy for human Blender users. Enabled by BlenderGym, we study how
  inference scaling techniques impact VLM's performance on graphics editing
  tasks. Notably, our findings reveal that the verifier used to guide the scaling
  of generation can itself be improved through inference scaling, complementing
  recent insights on inference scaling of LLM generation in coding and math
  tasks. We further show that inference compute is not uniformly effective and
  can be optimized by strategically distributing it between generation and
  verification.
paper_ai_summary: BlenderGym, a benchmark for 3D graphics editing, reveals that even advanced vision-language models struggle with tasks easily handled by human users and suggests optimizing computational resources between generation and verification.
paper_ai_keywords: vision-language models 3D graphics editing code-based 3D reconstruction BlenderGym inference scaling verifier generation verification