Language Model, Transformers, Deep Learning, Artificial Intelligence, Machine Learning.
regular dot-product attention using an approach called “Fast attention via positive orthogonal random features” FAVOR+. FAVOR+ combines low-rank approximation, matrix factorization, and matrix decomposition; then the space and time complexity becomes much more linear. Reformer Time: O(T log Td)