content.raw: 🎉 We're excited to introduce MemoryCode, a novel synthetic dataset designed to rigorously evaluate LLMs' ability to track and execute coding instructions across multiple sessions. MemoryCode simulates realistic workplace scenarios where a mentee (the LLM) receives coding instructions from a mentor amidst a stream of both relevant and irrelevant information. 
content.type: text
content.value: 🎉 We're excited to introduce MemoryCode, a novel synthetic dataset designed to rigorously evaluate LLMs' ability to track and execute coding instructions across multiple sessions. MemoryCode simulates realistic workplace scenarios where a mentee (the LLM) receives coding instructions from a mentor amidst a stream of both relevant and irrelevant information. 
content.type: new_line
content.type: new_line
content.raw: 💡 But what makes MemoryCode unique?! The combination of the following:
content.type: text
content.value: 💡 But what makes MemoryCode unique?! The combination of the following:
content.type: new_line
content.type: new_line
content.raw: ✅ Multi-Session Dialogue Histories: MemoryCode consists of chronological sequences of dialogues between a mentor and a mentee, mirroring real-world interactions between coworkers.
content.type: text
content.value: ✅ Multi-Session Dialogue Histories: MemoryCode consists of chronological sequences of dialogues between a mentor and a mentee, mirroring real-world interactions between coworkers.
content.type: new_line
content.type: new_line
content.raw: ✅ Interspersed Irrelevant Information: Critical instructions are deliberately interspersed with unrelated content, replicating the information overload common in office environments.
content.type: text
content.value: ✅ Interspersed Irrelevant Information: Critical instructions are deliberately interspersed with unrelated content, replicating the information overload common in office environments.
content.type: new_line
content.type: new_line
content.raw: ✅ Instruction Updates: Coding rules and conventions can be updated multiple times throughout the dialogue history, requiring LLMs to track and apply the most recent information.
content.type: text
content.value: ✅ Instruction Updates: Coding rules and conventions can be updated multiple times throughout the dialogue history, requiring LLMs to track and apply the most recent information.
content.type: new_line
content.type: new_line
content.raw: ✅ Prospective Memory: Unlike previous datasets that cue information retrieval, MemoryCode requires LLMs to spontaneously recall and apply relevant instructions without explicit prompts.
content.type: text
content.value: ✅ Prospective Memory: Unlike previous datasets that cue information retrieval, MemoryCode requires LLMs to spontaneously recall and apply relevant instructions without explicit prompts.
content.type: new_line
content.type: new_line
content.raw: ✅ Practical Task Execution: LLMs are evaluated on their ability to use the retrieved information to perform practical coding tasks, bridging the gap between information recall and real-world application.
content.type: text
content.value: ✅ Practical Task Execution: LLMs are evaluated on their ability to use the retrieved information to perform practical coding tasks, bridging the gap between information recall and real-world application.
content.type: new_line
content.type: new_line
content.raw: 📌 Our Findings
content.type: text
content.value: 📌 Our Findings
content.type: new_line
content.type: new_line
content.raw: 1️⃣ While even small models can handle isolated coding instructions, the performance of top-tier models like GPT-4o dramatically deteriorates when instructions are spread across multiple sessions.
content.type: text
content.value: 1️⃣ While even small models can handle isolated coding instructions, the performance of top-tier models like GPT-4o dramatically deteriorates when instructions are spread across multiple sessions.
content.type: new_line
content.type: new_line
content.raw: 2️⃣ This performance drop isn't simply due to the length of the context. Our analysis indicates that LLMs struggle to reason compositionally over sequences of instructions and updates. They have difficulty keeping track of which instructions are current and how to apply them.
content.type: text
content.value: 2️⃣ This performance drop isn't simply due to the length of the context. Our analysis indicates that LLMs struggle to reason compositionally over sequences of instructions and updates. They have difficulty keeping track of which instructions are current and how to apply them.
content.type: new_line
content.type: new_line
content.raw: 🔗 Paper: 
content.type: text
content.value: 🔗 Paper: 
content.label: From Tools to Teammates: Evaluating LLMs in Multi-Session Coding
    Interactions (2502.13791)
content.raw: https://huggingface.co/papers/2502.13791
content.id: 2502.13791
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/2502.13791
content.type: new_line
content.raw: 📦 Code: 
content.type: text
content.value: 📦 Code: 
content.href: https://github.com/for-ai/MemoryCode
content.raw: https://github.com/for-ai/MemoryCode
content.type: link
rawContent: 🎉 We're excited to introduce MemoryCode, a novel synthetic dataset designed to rigorously evaluate LLMs' ability to track and execute coding instructions across multiple sessions. MemoryCode simulates realistic workplace scenarios where a mentee (the LLM) receives coding instructions from a mentor amidst a stream of both relevant and irrelevant information. 
  
  💡 But what makes MemoryCode unique?! The combination of the following:
  
  ✅ Multi-Session Dialogue Histories: MemoryCode consists of chronological sequences of dialogues between a mentor and a mentee, mirroring real-world interactions between coworkers.
  
  ✅ Interspersed Irrelevant Information: Critical instructions are deliberately interspersed with unrelated content, replicating the information overload common in office environments.
  
  ✅ Instruction Updates: Coding rules and conventions can be updated multiple times throughout the dialogue history, requiring LLMs to track and apply the most recent information.
  
  ✅ Prospective Memory: Unlike previous datasets that cue information retrieval, MemoryCode requires LLMs to spontaneously recall and apply relevant instructions without explicit prompts.
  
  ✅ Practical Task Execution: LLMs are evaluated on their ability to use the retrieved information to perform practical coding tasks, bridging the gap between information recall and real-world application.
  
  📌 Our Findings
  
  1️⃣ While even small models can handle isolated coding instructions, the performance of top-tier models like GPT-4o dramatically deteriorates when instructions are spread across multiple sessions.
  
  2️⃣ This performance drop isn't simply due to the length of the context. Our analysis indicates that LLMs struggle to reason compositionally over sequences of instructions and updates. They have difficulty keeping track of which instructions are current and how to apply them.
  
  🔗 Paper: https://huggingface.co/papers/2502.13791
  📦 Code: https://github.com/for-ai/MemoryCode
reactions.reaction: 🔥
reactions.users: merterbak DjangoReinhardt707 John6666 Notnaton rdsm dantezxcd
reactions.reaction: ❤️
reactions.users: Jericho203 shetumohanto dantezxcd