fnlp 's Collections

Low Rank Sparse Attention

Open source weights of Lorsa modules introduced in "Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition".