Explain the approach and intuition behind self-attention in transformer models. In addition, provide information about sub-quadratic variants along with their strengths and weaknesses.
Self attention is a technique used to model interactions between different parts of an input sequence by computing weighted sums over those parts. It can be thought of as a generalization of dot products where instead of multiplying each element of two sequences together we multiply them with a weight matrix that captures how important they are for predicting the next token.

The main idea behind self-attention is to compute these weights using a query vector which selects a subset of tokens from the input sequence and a key vector which indicates what part of the selected tokens should contribute most to the computation. The output of this operation is then passed through another set of weights called the value matrix which computes the importance score of each token based on its interaction with all other tokens in the input sequence. This process is repeated multiple times until the entire input sequence has been processed.

One advantage of self-attention is that it allows us to capture long range dependencies within the input sequence without having to explicitly store or compute these relationships. Another benefit is that it provides a way to control the amount of attention given to specific parts of the input sequence while still capturing global structure. Finally, because it operates at the word level rather than character level, it's easier to apply to inputs such as text or audio data.