GPT-3 and Scaling Trends
Does anyone know of research into chatbot memory? I found this one: <a href="https:&#x2F;&#x2F;deepai.org&#x2F;publication&#x2F;a-proposal-for-intelligent-agents-with-episodic-memory" rel="nofollow">https:&#x2F;&#x2F;deepai.org&#x2F;publication&#x2F;a-proposal-for-intelligent-ag...</a><p>GPT-2 was a solution to a certain kind of problem: is it possible to throw out the idea of representing language while still having good performance on a variety of language tasks? But there doesn&#x27;t seem to be anything like that for &quot;memory,&quot; which seems separate and distinct from &quot;tasks.&quot;<p>In concrete terms, I&#x27;m interested in models that have memory in the sense of &quot;When you inference from the model, you leave a lasting impact on the model.&quot; Inferencing from the model should cause a change in the model parameters. Yet most models currently seem to assume that model parameters can be frozen without losing something essential to the end goal.<p>We&#x27;re very task-oriented. But none of these chat bots can remember my name, or anything about me, and it&#x27;s always bugged me. GPT-2 (and now GPT-3) punts the problem to a sufficiently clever programmer: just figure out how to encode all the &quot;memory&quot; into the context window, and then out pops the results you want. But that feels rather like arguing &quot;Just come up with a technique that works, and it will work.&quot; Perhaps it&#x27;s true, but not too helpful.<p>If you hear the same name a few times, you&#x27;ll remember it a long time, and start associating it with someone&#x27;s face. It seems like language models could do something similar. I don&#x27;t know precisely what; maybe someone here does.<p>You could designate part of the model as long-term memory, short-term, etc. Inferencing from the model could cause larger effects in the short-term area than the long-term area (equivalent to a higher learning rate).