title: WAFFLE: Multi-Modal Model for Automated Front-End Development
summary: Web development involves turning UI designs into functional webpages, which
  can be difficult for both beginners and experienced developers due to the
  complexity of HTML's hierarchical structures and styles. While Large Language
  Models (LLMs) have shown promise in generating source code, two major
  challenges persist in UI-to-HTML code generation: (1) effectively representing
  HTML's hierarchical structure for LLMs, and (2) bridging the gap between the
  visual nature of UI designs and the text-based format of HTML code. To tackle
  these challenges, we introduce Waffle, a new fine-tuning strategy that uses a
  structure-aware attention mechanism to improve LLMs' understanding of HTML's
  structure and a contrastive fine-tuning approach to align LLMs' understanding
  of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp
  (percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP,
  and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing
  benchmark Design2Code, outperforming current fine-tuning methods.
paper_authors._id: 671b2502320d7fb67e00dd7f
paper_authors.name: Shanchao Liang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-25T04:56:36.510Z
paper_authors._id: 6604b3c18191f5a7efa86eba
paper_authors.avatarUrl: /avatars/1ee0ae7e999f1fb3b068297eabb1bc8f.svg
paper_authors.fullname: Shanchao Liang
paper_authors.type: user
paper_authors.user: shanchao
paper_authors._id: 671b2502320d7fb67e00dd80
paper_authors.name: Nan Jiang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-25T05:33:56.094Z
paper_authors._id: 629e4ca2f2bda18349b6d330
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/629e4ca2f2bda18349b6d330/gSnGTLm2ugpINECylbuuQ.jpeg
paper_authors.fullname: Nan Jiang
paper_authors.type: user
paper_authors.user: jiang719
paper_authors._id: 671b2502320d7fb67e00dd81
paper_authors.name: Shangshu Qian
paper_authors._id: 671b2502320d7fb67e00dd82
paper_authors.name: Lin Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:04:02.989Z
paper_authors._id: 65831342f9c5cda913df366a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65831342f9c5cda913df366a/h7MLYt--shRYQj4-q5XmR.jpeg
paper_authors.fullname: Lin Tan
paper_authors.type: user
paper_authors.user: lin-tan
paper_title: WAFFLE: Multi-Modal Model for Automated Front-End Development
paper_summary: Web development involves turning UI designs into functional webpages, which
  can be difficult for both beginners and experienced developers due to the
  complexity of HTML's hierarchical structures and styles. While Large Language
  Models (LLMs) have shown promise in generating source code, two major
  challenges persist in UI-to-HTML code generation: (1) effectively representing
  HTML's hierarchical structure for LLMs, and (2) bridging the gap between the
  visual nature of UI designs and the text-based format of HTML code. To tackle
  these challenges, we introduce Waffle, a new fine-tuning strategy that uses a
  structure-aware attention mechanism to improve LLMs' understanding of HTML's
  structure and a contrastive fine-tuning approach to align LLMs' understanding
  of UI images and HTML code. Models fine-tuned with Waffle show up to 9.00 pp
  (percentage point) higher HTML match, 0.0982 higher CW-SSIM, 32.99 higher CLIP,
  and 27.12 pp higher LLEM on our new benchmark WebSight-Test and an existing
  benchmark Design2Code, outperforming current fine-tuning methods.
paper_ai_keywords: structure-aware attention mechanism contrastive fine-tuning approach HTML match CW-SSIM CLIP LLEM