WAFFLE: Multi-Modal Model for Automated Front-End Development
Web development involves turning UI designs into functional webpages, which
can be difficult for both beginners and experienced developers due to the
complexity of HTML's hierarchical structures and styles. While Large Language
Models (LLMs) have shown promise in generating source code, two major
challenges persist in UI-to-HTML code generation: (1) effectively representing
HTML's hierarchical structure for LLMs, and (2) bridging the gap between the
visual nature of UI designs and the text-based format of HTML code. To tackle
these challenges, we introduce Waffle, a new fine-tuning strategy that uses a
structure-aware attention mechanism to improve LLMs' understanding of HTML's
structure and a contrastive fine-tuning approach to align LLMs' understanding
of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp
(percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP,
and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing
benchmark Design2Code, outperforming current fine-tuning methods.
671b2502320d7fb67e00dd7f
Shanchao Liang
extracted_pending
2024-10-25T04:56:36.510Z
6604b3c18191f5a7efa86eba
/avatars/1ee0ae7e999f1fb3b068297eabb1bc8f.svg
Shanchao Liang
user
shanchao
671b2502320d7fb67e00dd80
Nan Jiang
extracted_confirmed
2024-10-25T05:33:56.094Z
629e4ca2f2bda18349b6d330
https://cdn-avatars.huggingface.co/v1/production/uploads/629e4ca2f2bda18349b6d330/gSnGTLm2ugpINECylbuuQ.jpeg
Nan Jiang
user
jiang719
671b2502320d7fb67e00dd81
Shangshu Qian
extracted_confirmed
2025-06-28T23:18:04.021Z
686065b147ff272201b331bc
/avatars/3512e5391c695cd0ca0036adafedb2f9.svg
Shangshu Qian
user
shangshu-qian
671b2502320d7fb67e00dd82
Lin Tan
claimed_verified
2024-10-28T11:04:02.989Z
65831342f9c5cda913df366a
https://cdn-avatars.huggingface.co/v1/production/uploads/65831342f9c5cda913df366a/h7MLYt--shRYQj4-q5XmR.jpeg
Lin Tan
user
lin-tan
WAFFLE: Multi-Modal Model for Automated Front-End Development
Web development involves turning UI designs into functional webpages, which
can be difficult for both beginners and experienced developers due to the
complexity of HTML's hierarchical structures and styles. While Large Language
Models (LLMs) have shown promise in generating source code, two major
challenges persist in UI-to-HTML code generation: (1) effectively representing
HTML's hierarchical structure for LLMs, and (2) bridging the gap between the
visual nature of UI designs and the text-based format of HTML code. To tackle
these challenges, we introduce Waffle, a new fine-tuning strategy that uses a
structure-aware attention mechanism to improve LLMs' understanding of HTML's
structure and a contrastive fine-tuning approach to align LLMs' understanding
of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp
(percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP,
and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing
benchmark Design2Code, outperforming current fine-tuning methods.
Waffle, a fine-tuning strategy with structure-aware attention and contrastive learning, improves LLMs' HTML code generation from UI designs, achieving better performance on benchmarks compared to existing methods.
Large Language Models UI-to-HTML code generation structure-aware attention mechanism contrastive fine-tuning HTML match CW-SSIM CLIP LLEM WebSight-Test Design2Code