title: Design2Code: How Far Are We From Automating Front-End Engineering?
summary: Generative AI has made rapid advancements in recent years, achieving
  unprecedented capabilities in multimodal understanding and code generation.
  This can enable a new paradigm of front-end development, in which multimodal
  LLMs might directly convert visual designs into code implementations. In this
  work, we formalize this as a Design2Code task and conduct comprehensive
  benchmarking. Specifically, we manually curate a benchmark of 484 diverse
  real-world webpages as test cases and develop a set of automatic evaluation
  metrics to assess how well current multimodal LLMs can generate the code
  implementations that directly render into the given reference webpages, given
  the screenshots as input. We also complement automatic metrics with
  comprehensive human evaluations. We develop a suite of multimodal prompting
  methods and show their effectiveness on GPT-4V and Gemini Pro Vision. We
  further finetune an open-source Design2Code-18B model that successfully matches
  the performance of Gemini Pro Vision. Both human evaluation and automatic
  metrics show that GPT-4V performs the best on this task compared to other
  models. Moreover, annotators think GPT-4V generated webpages can replace the
  original reference webpages in 49% of cases in terms of visual appearance and
  content; and perhaps surprisingly, in 64% of cases GPT-4V generated webpages
  are considered better than the original reference webpages. Our fine-grained
  break-down metrics indicate that open-source models mostly lag in recalling
  visual elements from the input webpages and in generating correct layout
  designs, while aspects like text content and coloring can be drastically
  improved with proper finetuning.
paper_authors._id: 65e7d0ff767bfc7d10ba6d18
paper_authors.name: Chenglei Si
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T09:32:07.414Z
paper_authors._id: 6144c0aa8a3e54bb3ecef8ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631895645405-noauth.jpeg
paper_authors.fullname: Chenglei Si
paper_authors.type: user
paper_authors.user: CLS
paper_authors._id: 65e7d0ff767bfc7d10ba6d19
paper_authors.name: Yanzhe Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T09:30:40.882Z
paper_authors._id: 61362307df63bbd1c2459c4c
paper_authors.avatarUrl: /avatars/28d3ba4b0fa00acb1eed623ab3620789.svg
paper_authors.fullname: Yanzhe Zhang
paper_authors.type: user
paper_authors.user: zyanzhe
paper_authors._id: 65e7d0ff767bfc7d10ba6d1a
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T09:30:48.105Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 65e7d0ff767bfc7d10ba6d1b
paper_authors.name: Ruibo Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-06T09:31:16.887Z
paper_authors._id: 5e91f1554957053f606489ff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5e91f1554957053f606489ff/eJty7TksRLwuta0deZTmt.jpeg
paper_authors.fullname: Ruibo Liu
paper_authors.type: user
paper_authors.user: agi-css
paper_authors._id: 65e7d0ff767bfc7d10ba6d1c
paper_authors.name: Diyi Yang
paper_title: Design2Code: How Far Are We From Automating Front-End Engineering?
paper_summary: Generative AI has made rapid advancements in recent years, achieving
  unprecedented capabilities in multimodal understanding and code generation.
  This can enable a new paradigm of front-end development, in which multimodal
  LLMs might directly convert visual designs into code implementations. In this
  work, we formalize this as a Design2Code task and conduct comprehensive
  benchmarking. Specifically, we manually curate a benchmark of 484 diverse
  real-world webpages as test cases and develop a set of automatic evaluation
  metrics to assess how well current multimodal LLMs can generate the code
  implementations that directly render into the given reference webpages, given
  the screenshots as input. We also complement automatic metrics with
  comprehensive human evaluations. We develop a suite of multimodal prompting
  methods and show their effectiveness on GPT-4V and Gemini Pro Vision. We
  further finetune an open-source Design2Code-18B model that successfully matches
  the performance of Gemini Pro Vision. Both human evaluation and automatic
  metrics show that GPT-4V performs the best on this task compared to other
  models. Moreover, annotators think GPT-4V generated webpages can replace the
  original reference webpages in 49% of cases in terms of visual appearance and
  content; and perhaps surprisingly, in 64% of cases GPT-4V generated webpages
  are considered better than the original reference webpages. Our fine-grained
  break-down metrics indicate that open-source models mostly lag in recalling
  visual elements from the input webpages and in generating correct layout
  designs, while aspects like text content and coloring can be drastically
  improved with proper finetuning.
paper_ai_summary: Evaluates the performance of multimodal LLMs, including GPT-4V and Gemini Pro Vision, on converting visual designs to code using a new Design2Code task and benchmarks.
paper_ai_keywords: multimodal LLMs Design2Code benchmarking automatic evaluation metrics human evaluations multimodal prompting GPT-4V Gemini Pro Vision fine-grained metrics visual elements layout designs text content coloring