Towards Data Science, Neural Networks, Deep Learning, Machine Learning, Naturallanguageprocessing.
networks almost exclusively use attention blocks. Attention helps to draw connections between any parts of the sequence, so long-range dependencies are not a problem anymore. With transformers, long-range dependencies have the same likelihood of being taken into account as any other short-range