AI’s Language Problem
Deep learning has succeeded tremendously with perception in domains that tolerate lots of noise (audio&#x2F;visual). Will those successes continue with <i>perception in domains that are not noisy</i> (language) and <i>inference&#x2F;control</i>, which the article touches on? I think it really is unclear whether those challenges will require fundamental developments or just more years of incremental improvement. If fundamental developments are needed, then the timeline for progress - which everyone in tech seems to be interested in - becomes much more indeterminate.<p>If you think about audio&#x2F;visual data, deep nets make sense: if you tweak a few pixel values in an image, or if you shift every pixel value by some amount, the image will still retain basically the same information. In this context, linearity (weighting values and summing them up) make sense. It&#x27;s not clear whether this makes sense in language. On the other hand, deep methods are state of the art on most NLP tasks, but their improvement over other methods isn&#x27;t the huge gap as in computer vision. And while we <i>know</i> there are tight similarities between lower-level visual features in deep nets and the initial layers of the visual cortex, the justification for deep learning in NLP is simpler and less specific: what I see is the fact that networks have huge capacity to fit to data and are deep (rely on a hierarchy of features). My guess is we may need a fundamental breakthrough in a newfangled hierarchical learning system that is better suited for language to “solve” NLP.<p>I think there are similar limitations with control and inference. When it comes to AlphaGo the deep learning component is responsible for estimating the value of the game state; the planning component is done with older methods. This is much more speculative, but when it comes to the work on Atari games, for example, I <i>suspect</i> that most of what is being learned (and solved) is perception of useful features from the raw game images. I wonder whether the features for deducing game state score are actually complex.<p>I think what I&#x27;m trying to say is that when we look at the success of deep learning, we have to separate out what part of that is due to the fact that deep learning is <i>the go-to</i> blackbox classifier, and what part of this is due to the systems we use actually being a good model for the problem. If the model isn&#x27;t good, does that model merely need to be tweaked from what we currently use, or does the model have to completely change?