Why Mastering Language Is So Difficult for AI
Language is a communication method evolved by intelligent beings, not a (primary) constituent of intelligence.  From neurology it&#x27;s pretty clear that the basic architecture of human minds is functional interconnected neural networks and not symbolic processing.  My belief is that world-modeling and prediction is the vast majority of what intelligence is, which is quite close to what the LLMs are doing.  World models can be in many representations (symbolic, logic gates, neural networks) but what matters is how accurate they are with respect to reality, and how well the model state is mapped from sensory input and back into real-world outputs.  Symbolic human language relies on each person&#x27;s internal world model and is learned by interacting with other humans who share a common language and similar enough world models, not the other way around (learning the world model as an aspect of the language itself).  Children learn which language inputs and outputs are beneficial and enjoyable to them using their native intelligence and can strengthen their world model with questions and answers that inform their model without having to directly experience what they are asking about.<p>People who don&#x27;t believe the LLMs have a world model are wrong because they are mistaking a physically weak world model for no world model.  GPT-3 doesn&#x27;t understand physics well enough to embed models of the referents of language into a unified model that has accurate gravity and motion dynamics, so it maintains a much more dreamlike model where objects exist in scenes and have relationships to each other but those relationships are governed by literary relationships instead of physical ones and so contradictions and superpositions and causality violations are allowed in the model.  As multimodal transformers like Gato get trained on more combined language and sensory input their world models will become much more physically and causally accurate which will be reflected in their accuracy on NLP tasks.