title: From Tools to Teammates: Evaluating LLMs in Multi-Session Coding
    Interactions
summary: Large Language Models (LLMs) are increasingly used in working environments
  for a wide range of tasks, excelling at solving individual problems in
  isolation. However, are they also able to effectively collaborate over
  long-term interactions? To investigate this, we introduce MemoryCode, a
  synthetic multi-session dataset designed to test LLMs' ability to track and
  execute simple coding instructions amid irrelevant information, simulating a
  realistic setting. While all the models we tested handle isolated instructions
  well, even the performance of state-of-the-art models like GPT-4o deteriorates
  when instructions are spread across sessions. Our analysis suggests this is due
  to their failure to retrieve and integrate information over long instruction
  chains. Our results highlight a fundamental limitation of current LLMs,
  restricting their ability to collaborate effectively in long interactions.
paper_authors._id: 67b7838bb41e5f760f8bd1b0
paper_authors.name: Nathanaël Carraz Rakotonirina
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T15:51:38.471Z
paper_authors._id: 6195d3199b7166aedc74247f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6195d3199b7166aedc74247f/8N5mUpSf3q5hRfMd8x70U.jpeg
paper_authors.fullname: Nathanaël Carraz Rakotonirina
paper_authors.type: user
paper_authors.user: nathanaelc
paper_authors._id: 67b7838bb41e5f760f8bd1b1
paper_authors.name: Mohammed Hamdy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T09:59:34.800Z
paper_authors._id: 62645f88c39850dc093d6105
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650745211725-noauth.png
paper_authors.fullname: Mohammed Hamdy
paper_authors.type: user
paper_authors.user: mmhamdy
paper_authors._id: 67b7838bb41e5f760f8bd1b2
paper_authors.name: Jon Ander Campos
paper_authors._id: 67b7838bb41e5f760f8bd1b3
paper_authors.name: Lucas Weber
paper_authors._id: 67b7838bb41e5f760f8bd1b4
paper_authors.name: Alberto Testoni
paper_authors._id: 67b7838bb41e5f760f8bd1b5
paper_authors.name: Marzieh Fadaee
paper_authors._id: 67b7838bb41e5f760f8bd1b6
paper_authors.name: Sandro Pezzelle
paper_authors._id: 67b7838bb41e5f760f8bd1b7
paper_authors.name: Marco Del Tredici
paper_title: From Tools to Teammates: Evaluating LLMs in Multi-Session Coding
    Interactions
paper_summary: Large Language Models (LLMs) are increasingly used in working environments
  for a wide range of tasks, excelling at solving individual problems in
  isolation. However, are they also able to effectively collaborate over
  long-term interactions? To investigate this, we introduce MemoryCode, a
  synthetic multi-session dataset designed to test LLMs' ability to track and
  execute simple coding instructions amid irrelevant information, simulating a
  realistic setting. While all the models we tested handle isolated instructions
  well, even the performance of state-of-the-art models like GPT-4o deteriorates
  when instructions are spread across sessions. Our analysis suggests this is due
  to their failure to retrieve and integrate information over long instruction
  chains. Our results highlight a fundamental limitation of current LLMs,
  restricting their ability to collaborate effectively in long interactions.