AI, Capacity, Attention Mechanism, Llm.
model considers 10 tokens at a time to make its predictions. Attention Heads and Soft Weights Attention heads are like different lenses through which the model can view tokens. Each head assigns a “soft weight” to tokens, indicating their importance in the context. These weights are “soft” because