Self Attention, Deep Learning, Transformers, Bert, Ml So Good.
the proper mixing is happening across tokens opens a pathway for trying out different approaches to mixing tokens. Due to practical applicability, smaller models are becoming prominent and this paper brings us a step closer to smaller transformers. References https://arxiv.org/abs/2105.03824 —