UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset
<|reference_start|>UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset: Automated UI evaluation can be beneficial for the design process; for example, to compare different UI designs, or conduct automated heuristic evaluation. LLM-based UI evaluation, in particular, holds the promise of generalizability to a wide variety of UI types and evaluation tasks. However, current LLM-based techniques do not yet match the performance of human evaluators. We hypothesize that automatic evaluation can be improved by collecting a targeted UI feedback dataset and then using this dataset to enhance the performance of general-purpose LLMs. We present a targeted dataset of 3,059 design critiques and quality ratings for 983 mobile UIs, collected from seven experienced designers. We carried out an in-depth analysis to characterize the dataset's features. We then applied this dataset to achieve a 55% performance gain in LLM-generated UI feedback via various few-shot and visual prompting techniques. We also discuss future applications of this dataset, including training a reward model for generative UI techniques, and fine-tuning a tool-agnostic multi-modal LLM that automates UI evaluation.<|reference_end|>
@article{duan2024uicrit:,
    title={UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset},
    author={Peitong Duan, Chin-yi Chen, Gang Li, Bjoern Hartmann, Yang Li},
    journal={arXiv preprint arXiv:2407.08850},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.08850},
    primaryClass={cs.HC cs.AI}
}