Self Attention, Deep Learning, Transformers, Bert, Ml So Good.
embedding input — one 1D DFT along the sequence dimension and one 1D DFT along the hidden dimension. Operation happening inside Fourier block (Source: Paper) DFT operations on input embeddings (Source: Author) The output of the Fourier transform is in the frequency domain. As we know the