It's not just statistics: GPT-4 does reason
Part 1: what are n-grams
Part 2: it&#x27;s using embeddings (but a lot of words without actually saying it)
Part 3: sufficiently trained NNs can sort things, which isn&#x27;t statistics<p>-----<p>I actually found some of the article interesting but not terribly convincing. Even though I consider these LLMs to be stochastic parrots, that isn&#x27;t to say they haven&#x27;t <i>learned</i> something during training, at least according to the colloquial meaning we typically ascribe to even lower models like MNIST classification. I&#x27;m even kind of okay with saying that it reasons about things in the same colloquial sense.<p>In a lot of ways, we just don&#x27;t have a good definition of what &#x27;reasoning&#x27; is. Is it just bad at reasoning because it&#x27;s input&#x2F;output&#x2F;modeling&#x2F;training is insufficient? Humans struggle to learn multiplication tables when we&#x27;re young. Are those humans not reasoning because they get the math wrong?<p>But there isn&#x27;t plasticity, there isn&#x27;t adaptability, it&#x27;s unclear to me that you can effectively inform it how to embed truly novel information - surely something that is <i>possible</i>, with some neurons existing for routing and activating other learned embeddings.<p>Anyway, interesting stuff.