Machine Learning, NLP, Computer Vision, Information Retrieval, Transformers.
which means that most of them are not used when computing a forward pass. Each Transformer layer a mixture-of-experts with hard routing at inference time, such that the number of operations per forward pass remains constant as you add experts, although the memory footprint and communication