title: Analysing the Residual Stream of Language Models Under Knowledge
    Conflicts
summary: Large language models (LLMs) can store a significant amount of factual
  knowledge in their parameters. However, their parametric knowledge may conflict
  with the information provided in the context. Such conflicts can lead to
  undesirable model behaviour, such as reliance on outdated or incorrect
  information. In this work, we investigate whether LLMs can identify knowledge
  conflicts and whether it is possible to know which source of knowledge the
  model will rely on by analysing the residual stream of the LLM. Through probing
  tasks, we find that LLMs can internally register the signal of knowledge
  conflict in the residual stream, which can be accurately detected by probing
  the intermediate model activations. This allows us to detect conflicts within
  the residual stream before generating the answers without modifying the input
  or model parameters. Moreover, we find that the residual stream shows
  significantly different patterns when the model relies on contextual knowledge
  versus parametric knowledge to resolve conflicts. This pattern can be employed
  to estimate the behaviour of LLMs when conflict happens and prevent unexpected
  answers before producing the answers. Our analysis offers insights into how
  LLMs internally manage knowledge conflicts and provides a foundation for
  developing methods to control the knowledge selection processes.
paper_authors._id: 6717ea90b954d4a1d5926d0e
paper_authors.name: Yu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-23T07:33:06.204Z
paper_authors._id: 654a500d9b8bd6406d431c0d
paper_authors.avatarUrl: /avatars/a6b76441bbc6f4b71d49c52e454c9ef7.svg
paper_authors.fullname: Yu Zhao
paper_authors.type: user
paper_authors.user: yuzhaouoe
paper_authors._id: 6717ea90b954d4a1d5926d0f
paper_authors.name: Xiaotang Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:42:42.571Z
paper_authors._id: 63d4698d1da8c28b7b8ab04a
paper_authors.avatarUrl: /avatars/73d4f49018db03a59e72b69bb2d1dd0c.svg
paper_authors.fullname: Xiaotang Du
paper_authors.type: user
paper_authors.user: acDante
paper_authors._id: 6717ea90b954d4a1d5926d10
paper_authors.name: Giwon Hong
paper_authors._id: 6717ea90b954d4a1d5926d11
paper_authors.name: Aryo Pradipta Gema
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:25:18.371Z
paper_authors._id: 644f895e23d7eb05ca695054
paper_authors.avatarUrl: /avatars/3fb04dd8544b403262bf98507de05453.svg
paper_authors.fullname: Aryo Pradipta Gema
paper_authors.type: user
paper_authors.user: aryopg
paper_authors._id: 6717ea90b954d4a1d5926d12
paper_authors.name: Alessio Devoto
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:55:16.409Z
paper_authors._id: 62180288d05d07ebf7de5f31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62180288d05d07ebf7de5f31/io2RsaRSjIDktQylXVvEB.jpeg
paper_authors.fullname: Alessio
paper_authors.type: user
paper_authors.user: alessiodevoto
paper_authors._id: 6717ea90b954d4a1d5926d13
paper_authors.name: Hongru Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:55:14.807Z
paper_authors._id: 65f906e5c3dbdcae83ff7aac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65f906e5c3dbdcae83ff7aac/mdjiVkLDJgJcGLwv0rMe4.jpeg
paper_authors.fullname: Hongru Wang
paper_authors.type: user
paper_authors.user: Merlin-Hongru
paper_authors._id: 6717ea90b954d4a1d5926d14
paper_authors.name: Xuanli He
paper_authors._id: 6717ea90b954d4a1d5926d15
paper_authors.name: Kam-Fai Wong
paper_authors._id: 6717ea90b954d4a1d5926d16
paper_authors.name: Pasquale Minervini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T15:51:35.008Z
paper_authors._id: 61001311e043e15c13412d30
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png
paper_authors.fullname: Pasquale Minervini
paper_authors.type: user
paper_authors.user: pminervini
paper_title: Analysing the Residual Stream of Language Models Under Knowledge
    Conflicts
paper_summary: Large language models (LLMs) can store a significant amount of factual
  knowledge in their parameters. However, their parametric knowledge may conflict
  with the information provided in the context. Such conflicts can lead to
  undesirable model behaviour, such as reliance on outdated or incorrect
  information. In this work, we investigate whether LLMs can identify knowledge
  conflicts and whether it is possible to know which source of knowledge the
  model will rely on by analysing the residual stream of the LLM. Through probing
  tasks, we find that LLMs can internally register the signal of knowledge
  conflict in the residual stream, which can be accurately detected by probing
  the intermediate model activations. This allows us to detect conflicts within
  the residual stream before generating the answers without modifying the input
  or model parameters. Moreover, we find that the residual stream shows
  significantly different patterns when the model relies on contextual knowledge
  versus parametric knowledge to resolve conflicts. This pattern can be employed
  to estimate the behaviour of LLMs when conflict happens and prevent unexpected
  answers before producing the answers. Our analysis offers insights into how
  LLMs internally manage knowledge conflicts and provides a foundation for
  developing methods to control the knowledge selection processes.
paper_ai_summary: LLMs can identify knowledge conflicts through their residual stream, allowing for detection and prediction of their reliance on contextual or parametric knowledge before generating responses.
paper_ai_keywords: LLMs knowledge conflicts residual stream intermediate model activations contextual knowledge parametric knowledge