Mass editing memory in a transformer
&gt;  GPT-3 predicts: Arneb is in the constellation of Aquila (incorrect - should be Lepus)<p>&gt; GPT-3 predicts: The current Vice President of the United States is named Mike Pence (obsolete)<p>These are qualitatively different things though.<p>Facts that are simply incorrect make sense to target and directly modify, but obsoleteness is a property of a fact, the subject transitions, the vice president is no longer current but was, it has a temporal property... I don&#x27;t know if LLMs can separately abstract that information from the subject in a way that is targetable - if it can&#x27;t, updating obsolete info feels like a perpetual task that grows in proportion to the breadth of learned information; whereas correcting facts that were always incorrect is proportional to the rate of additional learned knowledge multiplied by it&#x27;s accuracy.<p>The difference being that the work required to update facts is effectively constant over time, but the work required to update obsolete information (in this way) grows proportionally to the size of the model over time... assuming it makes sense to grow LLMs.