From Tools to Teammates: Evaluating LLMs in Multi-Session Coding
  Interactions
Large Language Models (LLMs) are increasingly used in working environments
for a wide range of tasks, excelling at solving individual problems in
isolation. However, are they also able to effectively collaborate over
long-term interactions? To investigate this, we introduce MemoryCode, a
synthetic multi-session dataset designed to test LLMs' ability to track and
execute simple coding instructions amid irrelevant information, simulating a
realistic setting. While all the models we tested handle isolated instructions
well, even the performance of state-of-the-art models like GPT-4o deteriorates
when instructions are spread across sessions. Our analysis suggests this is due
to their failure to retrieve and integrate information over long instruction
chains. Our results highlight a fundamental limitation of current LLMs,
restricting their ability to collaborate effectively in long interactions.
67b7838bb41e5f760f8bd1b0
Nathanaël Carraz Rakotonirina
claimed_verified
2025-02-24T15:51:38.471Z
6195d3199b7166aedc74247f
https://cdn-avatars.huggingface.co/v1/production/uploads/6195d3199b7166aedc74247f/8N5mUpSf3q5hRfMd8x70U.jpeg
Nathanaël Carraz Rakotonirina
user
nathanaelc
67b7838bb41e5f760f8bd1b1
Mohammed Hamdy
claimed_verified
2025-02-21T09:59:34.800Z
62645f88c39850dc093d6105
https://cdn-avatars.huggingface.co/v1/production/uploads/1650745211725-noauth.png
Mohammed Hamdy
user
mmhamdy
67b7838bb41e5f760f8bd1b2
Jon Ander Campos
67b7838bb41e5f760f8bd1b3
Lucas Weber
67b7838bb41e5f760f8bd1b4
Alberto Testoni
67b7838bb41e5f760f8bd1b5
Marzieh Fadaee
67b7838bb41e5f760f8bd1b6
Sandro Pezzelle
claimed_verified
2025-06-03T12:33:59.675Z
625523da03005d0910ba6776
/avatars/cfd43208990ffbcf71a37fd396b7f11c.svg
Sandro Pezzelle
user
sandropezzelle
67b7838bb41e5f760f8bd1b7
Marco Del Tredici
From Tools to Teammates: Evaluating LLMs in Multi-Session Coding
  Interactions
Large Language Models (LLMs) are increasingly used in working environments
for a wide range of tasks, excelling at solving individual problems in
isolation. However, are they also able to effectively collaborate over
long-term interactions? To investigate this, we introduce MemoryCode, a
synthetic multi-session dataset designed to test LLMs' ability to track and
execute simple coding instructions amid irrelevant information, simulating a
realistic setting. While all the models we tested handle isolated instructions
well, even the performance of state-of-the-art models like GPT-4o deteriorates
when instructions are spread across sessions. Our analysis suggests this is due
to their failure to retrieve and integrate information over long instruction
chains. Our results highlight a fundamental limitation of current LLMs,
restricting their ability to collaborate effectively in long interactions.
LLMs struggle to collaborate over long-term interactions, as demonstrated by their inability to track and integrate information across multiple sessions in a synthetic dataset.
Large Language Models LLMs MemoryCode synthetic multi-session dataset coding instructions long-term interactions GPT-4 information retrieval instruction chains