NLP, Artificial Intelligence, Data Science, Machine Learning, Neural Networks.
billion parameters, making it the biggest dense language model trained so far, 3x bigger than GPT-3. Bigger sparse models, such as mixture-of-experts, have been trained but it is unclear whether models built following this approach would have comparable parameter efficiency and generalization