Machine Learning, NLP, Computer Vision, Information Retrieval, Transformers.
overhead between computation nodes does increase. The paper has too much substance to condense in a couple of paragraphs, but I would stress that one of the most interesting findings is how scaling up Transformers via adding experts to each layer speeds up the learning substantially, keeping other