Machine Learning, NLP, Computer Vision, Information Retrieval, Transformers.
they use a pre-trained T5³ model. Similarly as with the Switch Transformers paper, there is a lot of substance to summarize in a paragraph. The two main pre-training tasks studied for the retriever are the Inverse Cloze Task¹ (ICT) and Masked salient spans²: Inverse Cloze Task (ICT): extracting