From RAGs to rich parameters: Probing how language models utilize
  external knowledge over parametric information for factual queries
Retrieval Augmented Generation (RAG) enriches the ability of language models
to reason using external context to augment responses for a given user prompt.
This approach has risen in popularity due to practical applications in various
applications of language models in search, question/answering, and chat-bots.
However, the exact nature of how this approach works isn't clearly understood.
In this paper, we mechanistically examine the RAG pipeline to highlight that
language models take shortcut and have a strong bias towards utilizing only the
context information to answer the question, while relying minimally on their
parametric memory. We probe this mechanistic behavior in language models with:
(i) Causal Mediation Analysis to show that the parametric memory is minimally
utilized when answering a question and (ii) Attention Contributions and
Knockouts to show that the last token residual stream do not get enriched from
the subject token in the question, but gets enriched from other informative
tokens in the context. We find this pronounced shortcut behaviour true across
both LLaMa and Phi family of models.
6672553486f6b3d126834783
Hitesh Wadhwa
claimed_verified
2024-06-19T07:37:38.560Z
64ebf8dff827684f65658073
/avatars/60cf2037ad7e1e2adb135660bd28204f.svg
Hitesh Wadhwa
user
wadhwahitesh
6672553486f6b3d126834784
Rahul Seetharaman
admin_assigned
2024-06-19T08:40:07.803Z
654295d34f89f533be6d4d2b
/avatars/4ac23b7ef93848232b46896add4f19a2.svg
Rahul Seetharaman
user
rahulseetharaman
6672553486f6b3d126834785
Somyaa Aggarwal
admin_assigned
2024-06-19T08:40:13.790Z
65df214be98700500d4cb0f4
/avatars/e370887a53c8740944b17c65a4f5eb07.svg
Somyaa Aggarwal
user
Somyaa27
6672553486f6b3d126834786
Reshmi Ghosh
extracted_confirmed
2024-06-19T12:49:23.836Z
63228cad95d6f717a8c8e1ec
/avatars/5fa606f63f841879af1a7617366d18b4.svg
Reshmi Ghosh
user
reshmighosh
6672553486f6b3d126834787
Samyadeep Basu
admin_assigned
2024-06-19T08:40:19.705Z
6384d891cafe4e4ac4060206
/avatars/45eb04bf6e0ed1a4daa14070411b693f.svg
Samyadeep
user
samyadeepbasu
6672553486f6b3d126834788
Soundararajan Srinivasan
6672553486f6b3d126834789
Wenlong Zhao
admin_assigned
2024-06-19T08:40:29.254Z
64b9e76820c3226970ab4abf
/avatars/1b361d83120f18f7cb950ba039991f6a.svg
WENLONG ZHAO
user
Warren7771818
6672553486f6b3d12683478a
Shreyas Chaudhari
6672553486f6b3d12683478b
Ehsan Aghazadeh
admin_assigned
2024-06-19T08:40:38.293Z
604f78723050a33ebb17ef63
https://cdn-avatars.huggingface.co/v1/production/uploads/1615820858280-noauth.jpeg
Ehsan Aghazadeh
user
EhsanAghazadeh
From RAGs to rich parameters: Probing how language models utilize
  external knowledge over parametric information for factual queries
Retrieval Augmented Generation (RAG) enriches the ability of language models
to reason using external context to augment responses for a given user prompt.
This approach has risen in popularity due to practical applications in various
applications of language models in search, question/answering, and chat-bots.
However, the exact nature of how this approach works isn't clearly understood.
In this paper, we mechanistically examine the RAG pipeline to highlight that
language models take shortcut and have a strong bias towards utilizing only the
context information to answer the question, while relying minimally on their
parametric memory. We probe this mechanistic behavior in language models with:
(i) Causal Mediation Analysis to show that the parametric memory is minimally
utilized when answering a question and (ii) Attention Contributions and
Knockouts to show that the last token residual stream do not get enriched from
the subject token in the question, but gets enriched from other informative
tokens in the context. We find this pronounced shortcut behaviour true across
both LLaMa and Phi family of models.
The study investigates the RAG framework and reveals that language models prefer using external context over their internal parametric memory, as demonstrated through causal mediation analysis and attention contributions.
Retrieval Augmented Generation RAG language models parametric memory causal mediation analysis attention contributions LLaMa Phi