Machine Learning, NLP, Computer Vision, Information Retrieval, Transformers.
the bigger ones which which outperform equivalent models trained from scratch; which is a learning paradigm that migh become dominant in the coming years. Performance gains as a function of sparse model parameters (left) and training steps (right). FLOPs at inference are equal for all cases