title: Better Embeddings with Coupled Adam
summary: Despite their remarkable capabilities, LLMs learn word representations that
  exhibit the undesirable yet poorly understood feature of anisotropy. In this
  paper, we argue that the second moment in Adam is a cause of anisotropic
  embeddings, and suggest a modified optimizer called Coupled Adam to mitigate
  the problem. Our experiments demonstrate that Coupled Adam significantly
  improves the quality of embeddings, while also leading to better upstream and
  downstream performance on large enough datasets.
paper_authors._id: 67b30311a2b3622dd42a51ff
paper_authors.name: Felix Stollenwerk
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:32:36.770Z
paper_authors._id: 66867e1675f10ce7ef96180e
paper_authors.avatarUrl: /avatars/ac85c00ba9d4dc48887b8864a0626743.svg
paper_authors.fullname: Felix Stollenwerk
paper_authors.type: user
paper_authors.user: flxst
paper_authors._id: 67b30311a2b3622dd42a5200
paper_authors.name: Tobias Stollenwerk
paper_title: Better Embeddings with Coupled Adam
paper_summary: Despite their remarkable capabilities, LLMs learn word representations that
  exhibit the undesirable yet poorly understood feature of anisotropy. In this
  paper, we argue that the second moment in Adam is a cause of anisotropic
  embeddings, and suggest a modified optimizer called Coupled Adam to mitigate
  the problem. Our experiments demonstrate that Coupled Adam significantly
  improves the quality of embeddings, while also leading to better upstream and
  downstream performance on large enough datasets.
paper_ai_summary: A modified optimizer named Coupled Adam addresses anisotropic embeddings in Large Language Models, improving embedding quality and task performance on large datasets.
paper_ai_keywords: LLMs anisotropy Adam second moment Coupled Adam embeddings upstream performance downstream performance