string edit distances (Perlitz et al., 2023).

5 Evaluation

We investigate the following questions. Q1. Does
CODEFUSION generate correct and diverse code?
Q2. How do different design decision impact per-
formance? Q3. How does the latent representation
evolve during the diffusion steps?

5.1 Performance and Diversity (Q1)
Table 1 summarizes performance in top-1, top-3
and top-5 settings for CODEFUSION and baselines.
In top-1, CODEFUSION performs on par with
or even better than (much larger) auto-regressive
models. For Python, only GPT-3 (175B) performs
better than CODEFUSION (75M). In top-3 and top-
5, CODEFUSION outperforms all baselines, consis-
tent with previous observations that auto-regressive
models with high top-1 performance sacrifice di-
versity in their generations (Poesia et al., 2022).
CODEFUSION