Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment
<|reference_start|>Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment: Large language and multimodal models have shown remarkable successes on various benchmarks focused on specific skills such as general-purpose programming, natural language understanding, math word problem-solving, and visual question answering. However, it is unclear how well these models perform on tasks that require a combination of these skills. In this paper, we curate a novel program synthesis benchmark based on the XLogoOnline visual programming environment. The benchmark comprises 85 real-world tasks from the Mini-level of the XLogoOnline environment, each requiring a combination of different skills such as spatial planning, basic programming, and logical reasoning. Our evaluation shows that current state-of-the-art models like GPT-4V and Llama3-70B struggle to solve these tasks, achieving only 20% and 2.35% success rates. Next, we develop a fine-tuning pipeline to boost the performance of models by leveraging a large-scale synthetic training dataset with over 80000 tasks. Moreover, we showcase how emulator-driven feedback can be used to design a curriculum over training data distribution. We showcase that a fine-tuned Llama3-8B drastically outperforms GPT-4V and Llama3-70B models, and provide an in-depth analysis of the models' expertise across different skill dimensions. We will publicly release the benchmark for future research on program synthesis in visual programming.<|reference_end|>
@article{wen2024program,
    title={Program Synthesis Benchmark for Visual Programming in XLogoOnline
  Environment},
    author={Chao Wen, Jacqueline Staub, Adish Singla},
    journal={arXiv preprint arXiv:2406.11334},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.11334},
    primaryClass={cs.AI}
}