approaches then select the vocabulary token with
the closest embedding. In the code domain, where
there are many syntactic and semantic constraints
between tokens, independently projecting embed-
dings back to tokens can yield invalid programs.

We propose CODEFUSION, a natural language
to code (NL-to-code) model that combines an
encoder-decoder architecture (Raffel et al., 2020)
with a diffusion process. The encoder maps the
NL into a continuous representation, which is used
by the diffusion model as an additional condition
for denoising random Gaussian noise input. To
generate syntactically correct code, we then feed
the denoised embeddings to a transformer decoder,
with full self-attention and cross attention with the
embedded utterance, to obtain probability distribu-
tions over code tokens. Finally, we select the token
with the highest probability at each index.
CODEFUSION