What is ChatGPT doing and why does it work?
The answer to this is: &quot;we don&#x27;t really know as its a very complex function automatically discovered by means of slow gradient descent, and we&#x27;re still finding out&quot;<p>Here are some of the fun things we&#x27;ve found out so far:<p>- GPT style language models try to build a model of the world: <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2210.13382" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2210.13382</a><p>- GPT style language models end up internally implementing a mini &quot;neural network training algorithm&quot; (gradient descent fine-tuning for given examples): <a href="https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2212.10559" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;abs&#x2F;2212.10559</a>