Towards Data Science, Neural Networks, Deep Learning, Machine Learning, Naturallanguageprocessing.
Self-Attention. 2.2 Query, Key, and Values The issue with Self-Attention is that nothing is being trained. But maybe if we add some trainable parameters, the network can then learn some patterns which give much better context. This trainable parameter can be a matrix whose values are trained. So