Language Model, Transformers, Deep Learning, Artificial Intelligence, Machine Learning.
which they claim to contribute to the model performance. Results show that after ~2.7B parameters, RetNet achieves lower perplexity and outperforms Transformer. Most of the results are reported based on the 6.7B model. RetNet is significantly better than Transformer at this scale in zero-shot,