OpenAI Codex
The "language models don't really understand anything" corner is getting smaller and smaller. In the last few months we've seen pretty definitive evidence that transformers can recombine concepts ([1], [2]) and do simple logical inference using contextual information ([3], "make the score font color visible"). I see no reason that this technology couldn't smoothly scale into human-level intelligence, yet lots of people seem to think it'll require a step change or is impossible.<p>That being said, robust systematic generalization is still a hard problem. But "achieve symbol grounding through tons of multimodal data" is looking more and more like the answer.<p>[1] <a href="https://openai.com/blog/dall-e/" rel="nofollow">https://openai.com/blog/dall-e/</a>
[2] <a href="https://distill.pub/2021/multimodal-neurons/" rel="nofollow">https://distill.pub/2021/multimodal-neurons/</a>
[3] <a href="https://openai.com/blog/openai-codex/" rel="nofollow">https://openai.com/blog/openai-codex/</a>