Microsoft Kosmos-1: A Multimodal Large Language Model
I like this feature they are working on<p><a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2212.10554" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2212.10554</a><p>as I&#x27;d say the most obvious limitation of today&#x27;s transformers is the limited attention window.  If you want ChatGPT to do a good job of summarizing a topic based on the literature the obvious thing is to feed a bunch of articles into it and ask it to summarize (how can you cite a paper you didn&#x27;t read?) and that requires looking at maybe 400,000 - 4,000,000 tokens.<p>Similarly there is a place for a word embedding,  a sentence embedding, a paragraph embedding, a chapter embedding, a book embedding, etc. but these have to be scalable and obviously the book embedding is bigger but I ought to be able to turn a query into a sentence embedding and somehow match it against larger document embeddings.