Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
expect is to see attention weights to move from right to left when given an input sequence. If so, this means our Transformer has learned a very simple grammar, which is just reading from right to left, and could generalize to more complex grammars when doing real-life language translation. Let’s