Answers.Body: <p>As you rightly noticed that normalized hypersphere embeddings are gaining popularity due to their apparent norm-bounded stability in training, especially in domains like contrastive/triplet learning and other metric learning where embeddings often represent specific meaningful distances between points and hypersphere constraint more effectively handles in-group similarity and out-group dissimilarity purely by the cosine similarity metric. Without normalization, vectors may vary in length rather than direction, making similarity metric more complicated and unstable in practice potentially.</p>
  <p>Euclidean embeddings remain standard in many foundational transformer models due to their powerful flexibility and implementation ease without similarity measure requirement such as in language modeling or other generative tasks. Though recently there's some interest to use hypersphere embedding for GPT models such as <a href="https://arxiv.org/html/2410.01131" rel="nofollow noreferrer">nGPT</a>.</p>
  <blockquote>
  <p>We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere... Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-09T01:33:37.627Z
Answers.LastActivityDate: 2024-11-09T01:33:37.627Z
Answers.UserDisplayName: cinch
Body: <p>There are many ways to create a &quot;deep embedding&quot;---by which I mean to project an input data point into a vector in a feature space, where this projection is learnt. To be useful, this vector should encode something semantic about the data.</p>
  <p>In various places I've seen the embedding referred to as an unconstrained value in Euclidean space:</p>
  <p><span class="math-container">$$z \in \mathbb{R}^D$$</span></p>
  <p>If it was truly unconstrained, there might be issues in training (e.g. divergence) so there will pretty much always be some kind of regularisation involved in learning the function.</p>
  <p>Alternatively the embedding is been L2-normalised before any further consideration</p>
  <p><span class="math-container">$$z' = z / |z|_2$$</span></p>
  <p>which is equivalent to saying the embedding represents data on the surface of a hypersphere</p>
  <p><span class="math-container">$$z' \in \textrm{SO}(D)$$</span></p>
  <p><strong>Which of these is common practice in current deep learning?</strong></p>
  <p>In other words, is it standard practice for people to L2-normalise their embeddings?</p>
  <p>There are various topological or geometric ways to argue for one way or the other. I'd like to understand what is common practice now. (And, if possible, <em>why</em> that choice is dominant.)</p>
  <p>Related literature: In the following examples of relevant papers, you'll see that hyperspherical embeddings seem to be common in some corners of AI work, but not all.</p>
  <ul>
  <li><a href="http://proceedings.mlr.press/v119/wang20k.html" rel="nofollow noreferrer">&quot;Understanding contrastive representation learning through alignment
  and uniformity on the hypersphere&quot;</a> (ICML 2020). This paper &quot;provides a comprehensive overview of the arguments for representation learning on the hypersphere&quot; including &quot;Spherical representations are associated with more stable training&quot;. The first sentence of this paper tells us: &quot;A vast number of recent empirical works learn representations with a unit L2 norm constraint, effectively restricting the output space to the unit hypersphere&quot;</li>
  <li><a href="https://arxiv.org/abs/1704.06369" rel="nofollow noreferrer">&quot;NormFace: L2 Hypersphere Embedding for Face Verification&quot;</a> (CVPR 2017) - I think this method was influential. It gives some geometric arguments for L2 normalising values. See also <a href="https://ai.stackexchange.com/questions/5438/how-is-the-constraint-fx-2-1-enforced-for-the-embedding-fx-in-the">FaceNet mentioned in this related question</a>.</li>
  <li><a href="https://proceedings.neurips.cc/paper/2020/hash/d9812f756d0df06c7381945d2e2c7d4b-Abstract.html" rel="nofollow noreferrer">&quot;Deep Metric Learning with Spherical Embedding&quot;</a> (NeurIPS 2020) -- even on page 1 it's already clear that they're assuming that deep embeddings &quot;should&quot; lie on a hypersphere.</li>
  </ul>
  <p>...and yet, see these very recent papers:</p>
  <ul>
  <li>In this <a href="https://icml.cc/virtual/2024/38896" rel="nofollow noreferrer">ICML 2024 poster</a> the authors claim that &quot;embeddings learning is mostly done in Euclidean space&quot; and they consider hyperspheres instead.</li>
  <li><a href="https://arxiv.org/abs/2410.01131" rel="nofollow noreferrer">&quot;nGPT: Normalized Transformer with Representation Learning on the Hypersphere&quot;</a> (NVIDIA, 2024). Claiming some novelty in the way they use normed values throughout. -- They specifically point out that &quot;the norms of embedding vectors in the original Transformer are unconstrained&quot;, thus I guess this is true for most transformers, as a result of the foundational work?</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-08T10:01:50.687Z
Comments.Text: Some models like Resnet use batch normalization which maps the data to Euclidean space but constrains the data variance.
Comments.UserDisplayName: Curious
Tags: <embeddings><geometric-deep-learning><representation-learning><topology>
Title: Deep embeddings: which is common practice, Euclidean space or normalised (hypersphere)?
UserDisplayName: Dan Stowell
ThreadText: # Deep embeddings: which is common practice, Euclidean space or normalised (hypersphere)?
  
  (Asked by: username_0 on 2024-11-08)
  
  There are many ways to create a "deep embedding"---by which I mean to project an input data point into a vector in a feature space, where this projection is learnt. To be useful, this vector should encode something semantic about the data.
  
  In various places I've seen the embedding referred to as an unconstrained value in Euclidean space:
  
  $$z \in \mathbb{R}^D$$
  
  If it was truly unconstrained, there might be issues in training (e.g. divergence) so there will pretty much always be some kind of regularisation involved in learning the function.
  
  Alternatively the embedding is been L2-normalised before any further consideration
  
  $$z' = z / |z|\_2$$
  
  which is equivalent to saying the embedding represents data on the surface of a hypersphere
  
  $$z' \in \textrm{SO}(D)$$
  
  **Which of these is common practice in current deep learning?**
  
  In other words, is it standard practice for people to L2-normalise their embeddings?
  
  There are various topological or geometric ways to argue for one way or the other. I'd like to understand what is common practice now. (And, if possible, *why* that choice is dominant.)
  
  Related literature: In the following examples of relevant papers, you'll see that hyperspherical embeddings seem to be common in some corners of AI work, but not all.
  
  * ["Understanding contrastive representation learning through alignment
    and uniformity on the hypersphere"](http://proceedings.mlr.press/v119/wang20k.html) (ICML 2020). This paper "provides a comprehensive overview of the arguments for representation learning on the hypersphere" including "Spherical representations are associated with more stable training". The first sentence of this paper tells us: "A vast number of recent empirical works learn representations with a unit L2 norm constraint, effectively restricting the output space to the unit hypersphere"
  * ["NormFace: L2 Hypersphere Embedding for Face Verification"](https://arxiv.org/abs/1704.06369) (CVPR 2017) - I think this method was influential. It gives some geometric arguments for L2 normalising values. See also [FaceNet mentioned in this related question](https://ai.stackexchange.com/questions/5438/how-is-the-constraint-fx-2-1-enforced-for-the-embedding-fx-in-the).
  * ["Deep Metric Learning with Spherical Embedding"](https://proceedings.neurips.cc/paper/2020/hash/d9812f756d0df06c7381945d2e2c7d4b-Abstract.html) (NeurIPS 2020) -- even on page 1 it's already clear that they're assuming that deep embeddings "should" lie on a hypersphere.
  
  ...and yet, see these very recent papers:
  
  * In this [ICML 2024 poster](https://icml.cc/virtual/2024/38896) the authors claim that "embeddings learning is mostly done in Euclidean space" and they consider hyperspheres instead.
  * ["nGPT: Normalized Transformer with Representation Learning on the Hypersphere"](https://arxiv.org/abs/2410.01131) (NVIDIA, 2024). Claiming some novelty in the way they use normed values throughout. -- They specifically point out that "the norms of embedding vectors in the original Transformer are unconstrained", thus I guess this is true for most transformers, as a result of the foundational work?
  
  **Tags:** embeddings, geometric-deep-learning, representation-learning, topology
  > **Comments:**
  > **username_1**: Some models like Resnet use batch normalization which maps the data to Euclidean space but constrains the data variance.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-11-09. Score: 1 | Accepted answer
  
  As you rightly noticed that normalized hypersphere embeddings are gaining popularity due to their apparent norm-bounded stability in training, especially in domains like contrastive/triplet learning and other metric learning where embeddings often represent specific meaningful distances between points and hypersphere constraint more effectively handles in-group similarity and out-group dissimilarity purely by the cosine similarity metric. Without normalization, vectors may vary in length rather than direction, making similarity metric more complicated and unstable in practice potentially.
  
  Euclidean embeddings remain standard in many foundational transformer models due to their powerful flexibility and implementation ease without similarity measure requirement such as in language modeling or other generative tasks. Though recently there's some interest to use hypersphere embedding for GPT models such as [nGPT](https://arxiv.org/html/2410.01131).
  
  > We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere... Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length.
  
  
  ---
  
  