Recursively summarizing enables long-term dialogue memory in LLMs
To my intuition, all of these ways of building memory in “text space” seem super hacky.It seems intuitive to me that memory would be best stored in dense embedding space that can preserve full semantic meaning for the model rather than as some hacked on process of continually regenerating summaries.And similarly, the model needs to be trained in a setting where it is aware of the memory and how to use it. Preferably that would be from the very beginning (ie. the train on text).