GPT-4 can create its own shorthand and recreate the original text
If this is the case and we assume there is no state shared between the summarization phase and the expansion phase, doesn&#x27;t this suggest that this is just a giant &quot;over-fit&quot; ml model, more like a compression algorithm (a very inefficient one), and is incapable of any original thought etc? This is my hypothesis. I mean another giant downside of the LLMs today is the fact that they are incapable of learning without throwing massive amounts of money and compute at the problem which I assume would be retraining ALL the weights from scratch. You could retrain part of the weights but that would no doubt negatively affect &quot;unconnected&quot; (in a human sense) parts of the model. Is this even sustainable at all and will it ever be?