Memory Integration
#1
by asphalteater - opened
Integrating memory into a model architecture is a highly constructive endeavor indeed. There’s just one concern of mine - would compressing the memory into a fixed-size parameter matrix lead to excessive information loss and potentially prevents the accurate retention of long context? What's more, it seems to me that the memory ops described in the paper as a sequential structure—reminiscent of an LSTM—which raises the question: in short contexts, does this approach incur longer inference times compared to models like GPT or Claude?