Artificial Intelligence, Deep Learning, Machine Learning, Neural Networks, Math.
and every element is equally relevant: As n (sequence length) increases, the sum of each row in the matrix QK^T (before applying the softmax) increases due to more terms being added, assuming non-zero vectors and general relevance among them. This can lead to a scenario where the impact of any