Figure 5: Figure showing the various stages of denoising in CODEFUSION on an example from the Python
benchmarks where CODEFUSION fails. CODEFUSION starts from pure noise and gradually denoises to generate
the target code. The final generation is incorrect here as the correct function name is os.removedirs and also this
function only removes empty directories while the user wanted to remove directory with files.

N(cid:89)

the benchmarks was found to be 2318 milliseconds.
The average GPU memory used was 928 Mega
Bytes and the maximum GPU memory used was
1042 Mega Bytes.

D Background
D.1 Transformer based Sequence Generation
Transformer based language models (Vaswani et al.,
2017) are conditional generative models imple-
mented through auto-regressive (AR) decoding.
These models predict the likelihood of the target
token yt using the conditional input encoding and
previously generated tokens y1, y2,··· , yt−1. The
likelihood of the generated sequence is given by:

P(y|x) =
CODEFUSION