Design2Code: How Far Are We From Automating Front-End Engineering?
Generative AI has made rapid advancements in recent years, achieving
unprecedented capabilities in multimodal understanding and code generation.
This can enable a new paradigm of front-end development, in which multimodal
LLMs might directly convert visual designs into code implementations. In this
work, we formalize this as a Design2Code task and conduct comprehensive
benchmarking. Specifically, we manually curate a benchmark of 484 diverse
real-world webpages as test cases and develop a set of automatic evaluation
metrics to assess how well current multimodal LLMs can generate the code
implementations that directly render into the given reference webpages, given
the screenshots as input. We also complement automatic metrics with
comprehensive human evaluations. We develop a suite of multimodal prompting
methods and show their effectiveness on GPT-4V and Gemini Pro Vision. We
further finetune an open-source Design2Code-18B model that successfully matches
the performance of Gemini Pro Vision. Both human evaluation and automatic
metrics show that GPT-4V performs the best on this task compared to other
models. Moreover, annotators think GPT-4V generated webpages can replace the
original reference webpages in 49% of cases in terms of visual appearance and
content; and perhaps surprisingly, in 64% of cases GPT-4V generated webpages
are considered better than the original reference webpages. Our fine-grained
break-down metrics indicate that open-source models mostly lag in recalling
visual elements from the input webpages and in generating correct layout
designs, while aspects like text content and coloring can be drastically
improved with proper finetuning.
65e7d0ff767bfc7d10ba6d18
Chenglei Si
admin_assigned
2024-03-06T09:32:07.414Z
6144c0aa8a3e54bb3ecef8ea
https://cdn-avatars.huggingface.co/v1/production/uploads/1631895645405-noauth.jpeg
Chenglei Si
user
CLS
65e7d0ff767bfc7d10ba6d19
Yanzhe Zhang
admin_assigned
2024-03-06T09:30:40.882Z
61362307df63bbd1c2459c4c
/avatars/28d3ba4b0fa00acb1eed623ab3620789.svg
Yanzhe Zhang
user
zyanzhe
65e7d0ff767bfc7d10ba6d1a
Zhengyuan Yang
admin_assigned
2024-03-06T09:30:48.105Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
65e7d0ff767bfc7d10ba6d1b
Ruibo Liu
admin_assigned
2024-03-06T09:31:16.887Z
5e91f1554957053f606489ff
https://cdn-avatars.huggingface.co/v1/production/uploads/5e91f1554957053f606489ff/eJty7TksRLwuta0deZTmt.jpeg
Ruibo Liu
user
agi-css
65e7d0ff767bfc7d10ba6d1c
Diyi Yang
Design2Code: How Far Are We From Automating Front-End Engineering?
Generative AI has made rapid advancements in recent years, achieving
unprecedented capabilities in multimodal understanding and code generation.
This can enable a new paradigm of front-end development, in which multimodal
LLMs might directly convert visual designs into code implementations. In this
work, we formalize this as a Design2Code task and conduct comprehensive
benchmarking. Specifically, we manually curate a benchmark of 484 diverse
real-world webpages as test cases and develop a set of automatic evaluation
metrics to assess how well current multimodal LLMs can generate the code
implementations that directly render into the given reference webpages, given
the screenshots as input. We also complement automatic metrics with
comprehensive human evaluations. We develop a suite of multimodal prompting
methods and show their effectiveness on GPT-4V and Gemini Pro Vision. We
further finetune an open-source Design2Code-18B model that successfully matches
the performance of Gemini Pro Vision. Both human evaluation and automatic
metrics show that GPT-4V performs the best on this task compared to other
models. Moreover, annotators think GPT-4V generated webpages can replace the
original reference webpages in 49% of cases in terms of visual appearance and
content; and perhaps surprisingly, in 64% of cases GPT-4V generated webpages
are considered better than the original reference webpages. Our fine-grained
break-down metrics indicate that open-source models mostly lag in recalling
visual elements from the input webpages and in generating correct layout
designs, while aspects like text content and coloring can be drastically
improved with proper finetuning.
Evaluates the performance of multimodal LLMs, including GPT-4V and Gemini Pro Vision, on converting visual designs to code using a new Design2Code task and benchmarks.
multimodal LLMs Design2Code benchmarking automatic evaluation metrics human evaluations multimodal prompting GPT-4V Gemini Pro Vision fine-grained metrics visual elements layout designs text content coloring