4.1 Benchmarks
We evaluate CODEFUSION on NL-to-code for lan-
guages with varying complexity: Python, Bash,
and conditional formatting (CF) rules in Microsoft
Excel. The CoNaLa dataset (Yin et al., 2018)
for Python consists of complex, multi-statement
StackOverflow code snippets and associated NL
questions. The Bash dataset (Lin et al., 2018) has
complex, single-line Bash commands annotated
with NL descriptions. The CF dataset (Singh et al.,
2022) consists of Excel CF rules, which are single
line programs of low complexity, annotated with
NL. These benchmarks (and pre-training data—see
next section) are made publicly available.1
CODEFUSION