Better Embeddings with Coupled Adam
Despite their remarkable capabilities, LLMs learn word representations that
exhibit the undesirable yet poorly understood feature of anisotropy. In this
paper, we argue that the second moment in Adam is a cause of anisotropic
embeddings, and suggest a modified optimizer called Coupled Adam to mitigate
the problem. Our experiments demonstrate that Coupled Adam significantly
improves the quality of embeddings, while also leading to better upstream and
downstream performance on large enough datasets.
67b30311a2b3622dd42a51ff
Felix Stollenwerk
claimed_verified
2025-02-18T09:32:36.770Z
66867e1675f10ce7ef96180e
/avatars/ac85c00ba9d4dc48887b8864a0626743.svg
Felix Stollenwerk
user
flxst
67b30311a2b3622dd42a5200
Tobias Stollenwerk
Better Embeddings with Coupled Adam
Despite their remarkable capabilities, LLMs learn word representations that
exhibit the undesirable yet poorly understood feature of anisotropy. In this
paper, we argue that the second moment in Adam is a cause of anisotropic
embeddings, and suggest a modified optimizer called Coupled Adam to mitigate
the problem. Our experiments demonstrate that Coupled Adam significantly
improves the quality of embeddings, while also leading to better upstream and
downstream performance on large enough datasets.
A modified optimizer named Coupled Adam addresses anisotropic embeddings in Large Language Models, improving embedding quality and task performance on large datasets.
LLMs anisotropy Adam second moment Coupled Adam embeddings upstream performance downstream performance