Language Model, Transformers, Deep Learning, Artificial Intelligence, Machine Learning.
vanilla Transformer model by a large margin in perplexity and computation. They were able to train LONGNET with 32k context size whereas the Transformer only 16k. My 2 cents: Consider using LONGNET when processing or streaming long sequences. MegaByte Time: O(T ^ (4/3) d) Space: O(T log Td) 📎 Paper