title: BlenderAlchemy: Editing 3D Graphics with Vision-Language Models
summary: Graphics design is important for various applications, including movie
  production and game design. To create a high-quality scene, designers usually
  need to spend hours in software like Blender, in which they might need to
  interleave and repeat operations, such as connecting material nodes, hundreds
  of times. Moreover, slightly different design goals may require completely
  different sequences, making automation difficult. In this paper, we propose a
  system that leverages Vision-Language Models (VLMs), like GPT-4V, to
  intelligently search the design action space to arrive at an answer that can
  satisfy a user's intent. Specifically, we design a vision-based edit generator
  and state evaluator to work together to find the correct sequence of actions to
  achieve the goal. Inspired by the role of visual imagination in the human
  design process, we supplement the visual reasoning capabilities of VLMs with
  "imagined" reference images from image-generation models, providing visual
  grounding of abstract language descriptions. In this paper, we provide
  empirical evidence suggesting our system can produce simple but tedious Blender
  editing sequences for tasks such as editing procedural materials from text
  and/or reference images, as well as adjusting lighting configurations for
  product renderings in complex scenes.
paper_authors._id: 663069379edc222ceeaea07e
paper_authors.name: Ian Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:24:29.899Z
paper_authors._id: 639ce51c66106be14370f510
paper_authors.avatarUrl: /avatars/74f68f5951fff9c02c7607ff197812aa.svg
paper_authors.fullname: Ian Huang
paper_authors.type: user
paper_authors.user: ianhuang
paper_authors._id: 663069379edc222ceeaea07f
paper_authors.name: Guandao Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-30T08:24:36.330Z
paper_authors._id: 64547ca006728ff79a38e7a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64547ca006728ff79a38e7a5/xfTVQXMSfRv-IbljFpswQ.jpeg
paper_authors.fullname: Guandao Yang
paper_authors.type: user
paper_authors.user: guandao
paper_authors._id: 663069379edc222ceeaea080
paper_authors.name: Leonidas Guibas
paper_title: BlenderAlchemy: Editing 3D Graphics with Vision-Language Models
paper_summary: Graphics design is important for various applications, including movie
  production and game design. To create a high-quality scene, designers usually
  need to spend hours in software like Blender, in which they might need to
  interleave and repeat operations, such as connecting material nodes, hundreds
  of times. Moreover, slightly different design goals may require completely
  different sequences, making automation difficult. In this paper, we propose a
  system that leverages Vision-Language Models (VLMs), like GPT-4V, to
  intelligently search the design action space to arrive at an answer that can
  satisfy a user's intent. Specifically, we design a vision-based edit generator
  and state evaluator to work together to find the correct sequence of actions to
  achieve the goal. Inspired by the role of visual imagination in the human
  design process, we supplement the visual reasoning capabilities of VLMs with
  "imagined" reference images from image-generation models, providing visual
  grounding of abstract language descriptions. In this paper, we provide
  empirical evidence suggesting our system can produce simple but tedious Blender
  editing sequences for tasks such as editing procedural materials from text
  and/or reference images, as well as adjusting lighting configurations for
  product renderings in complex scenes.
paper_ai_keywords: Vision-Language Models (VLMs) GPT-4V vision-based edit generator state evaluator imagined reference images image-generation models procedural materials product renderings