Turing-NLG: A 17B-parameter language model
People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP, and with it, the ability to create, synthesize, and understand content, will change the internet.<p>More than that, I think NLP will unlock new ways of interacting with computers. Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world.<p>Edit:<p>Adding this to give more technical context. I think most people don’t know where the line is currently between what possible, and what’s not, but also what we are on the cusp of. And we are on the cusp of a lot.<p>A quick explanation of one area is here:<p>Basically, transformer models are the best for NLP. They use something called attention based mechanisms, which allows the model to draw correlations between pieces of text&#x2F;tokens that are far apart. The issue is that this is an O(n^2) operation. So the model is bounded by the context window, which is currently mostly at 512 tokens, and is thus, bounded in how much it can understand.
Recent innovations, and further study, will broaden the context window, and thus unlock better reading comprehension and context understanding.
For instance, the ability to answer a question using a piece of text is mostly stuck at just finding one paragraph. The future will see models that can find multiple different paragraphs, understand how they relate, pull the relevant information, and synthesize it. This sounds like a minor step forwards, but its important.
This will unlock better conversational abilities, but also, better ways to understand how different pieces of textual information relate. The scattershot of information across the internet can go away. Computers can better understand context to act on human intention through language, unlocking the ability to handle ambiguity. This will change the internet.<p>Again to empathize, these models only started showing up in 2017! The progress has been rapid.