title: User-LLM: Efficient LLM Contextualization with User Embeddings
summary: Large language models (LLMs) have revolutionized natural language processing.
  However, effectively incorporating complex and potentially noisy user
  interaction data remains a challenge. To address this, we propose User-LLM, a
  novel framework that leverages user embeddings to contextualize LLMs. These
  embeddings, distilled from diverse user interactions using self-supervised
  pretraining, capture latent user preferences and their evolution over time. We
  integrate these user embeddings with LLMs through cross-attention and
  soft-prompting, enabling LLMs to dynamically adapt to user context. Our
  comprehensive experiments on MovieLens, Amazon Review, and Google Local Review
  datasets demonstrate significant performance gains across various tasks.
  Notably, our approach outperforms text-prompt-based contextualization on long
  sequence tasks and tasks that require deep user understanding while being
  computationally efficient. We further incorporate Perceiver layers to
  streamline the integration between user encoders and LLMs, reducing
  computational demands.
paper_authors._id: 65d6cc965e971572da052bf5
paper_authors.name: Lin Ning
paper_authors._id: 65d6cc965e971572da052bf6
paper_authors.name: Luyang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-24T14:36:58.415Z
paper_authors._id: 65d8fa4e16cebdd689fa5587
paper_authors.avatarUrl: /avatars/9646f22ae16653af47ea38af0839ae7b.svg
paper_authors.fullname: Luyang Liu
paper_authors.type: user
paper_authors.user: luyangl
paper_authors._id: 65d6cc965e971572da052bf7
paper_authors.name: Jiaxing Wu
paper_authors._id: 65d6cc965e971572da052bf8
paper_authors.name: Neo Wu
paper_authors._id: 65d6cc965e971572da052bf9
paper_authors.name: Devora Berlowitz
paper_authors._id: 65d6cc965e971572da052bfa
paper_authors.name: Sushant Prakash
paper_authors._id: 65d6cc965e971572da052bfb
paper_authors.name: Bradley Green
paper_authors._id: 65d6cc965e971572da052bfc
paper_authors.name: Shawn O'Banion
paper_authors._id: 65d6cc965e971572da052bfd
paper_authors.name: Jun Xie
paper_title: User-LLM: Efficient LLM Contextualization with User Embeddings
paper_summary: Large language models (LLMs) have revolutionized natural language processing.
  However, effectively incorporating complex and potentially noisy user
  interaction data remains a challenge. To address this, we propose User-LLM, a
  novel framework that leverages user embeddings to contextualize LLMs. These
  embeddings, distilled from diverse user interactions using self-supervised
  pretraining, capture latent user preferences and their evolution over time. We
  integrate these user embeddings with LLMs through cross-attention and
  soft-prompting, enabling LLMs to dynamically adapt to user context. Our
  comprehensive experiments on MovieLens, Amazon Review, and Google Local Review
  datasets demonstrate significant performance gains across various tasks.
  Notably, our approach outperforms text-prompt-based contextualization on long
  sequence tasks and tasks that require deep user understanding while being
  computationally efficient. We further incorporate Perceiver layers to
  streamline the integration between user encoders and LLMs, reducing
  computational demands.