Answers.Body: <p>Orthogonality ensures that the representations for distinct concepts are minimally correlated even if the inputs are similar at a surface level. For example, the orthogonality between contextual embeddings of &quot;bank&quot; as a financial institution and &quot;bank&quot; as the side of a river makes them totally distinct and easily separable. Thus orthogonality minimizes feature interference between different data during training which means that when the model learns something <em>new</em>, the changes made to its parameters have less impact on <em>unrelated</em> areas of knowledge thus stabilizing and aiding NLP tasks generalization. Furthermore orthogonality tends to bring sparse representations where only a few dimensions are significantly activated for any given input, thus improving optimization efficiency in terms of training convergence.</p>
  <p>The larger the scale, the more it can leverage orthogonality to store and process diverse concepts efficiently without seeming redundancy in the sense of either linguistic syntax or semantics, enabling better disentangling of features which is crucial for capturing nuanced relationships in language. For the analogy task where &quot;king - man + woman = queen&quot;, orthogonality ensures that
  the &quot;gender&quot; axis is distinct from the &quot;royalty&quot; axis and operations like subtraction and addition correspond to meaningful transformations in these orthogonal dimensions. Therefore orthogonality in high-dimensional spaces is one of the key reasons why scaling LLMs often leads to disproportionately better performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-23T05:56:06.093Z
Answers.LastActivityDate: 2024-11-23T06:44:36.017Z
Answers.LastEditDate: 2024-11-23T06:44:36.017Z
Answers.UserDisplayName: cinch
Body: <p>In <a href="https://www.youtube.com/watch?v=piJkuavhV50" rel="nofollow noreferrer">this video</a>, it is mentioned at 27:10 that as we go to higher and higher dimensional spaces, the probability of two random vectors being orthogonal goes to <span class="math-container">$1$</span>. And also that this might explain why large language models perform well only at scale.</p>
  <p>How does the orthogonality fact help large language models?</p>
  
Tags: <large-language-models>
Title: How do high dimensional vectors becoming perpendicular help explain why LLMs perform better at scale?
UserDisplayName: Rohit Pandey
ThreadText: # How do high dimensional vectors becoming perpendicular help explain why LLMs perform better at scale?
  
  (Asked by: username_0 on 2024-11-23)
  
  In [this video](https://www.youtube.com/watch?v=piJkuavhV50), it is mentioned at 27:10 that as we go to higher and higher dimensional spaces, the probability of two random vectors being orthogonal goes to $1$. And also that this might explain why large language models perform well only at scale.
  
  How does the orthogonality fact help large language models?
  
  **Tags:** large-language-models
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-23. Score: 4
  
  Orthogonality ensures that the representations for distinct concepts are minimally correlated even if the inputs are similar at a surface level. For example, the orthogonality between contextual embeddings of "bank" as a financial institution and "bank" as the side of a river makes them totally distinct and easily separable. Thus orthogonality minimizes feature interference between different data during training which means that when the model learns something *new*, the changes made to its parameters have less impact on *unrelated* areas of knowledge thus stabilizing and aiding NLP tasks generalization. Furthermore orthogonality tends to bring sparse representations where only a few dimensions are significantly activated for any given input, thus improving optimization efficiency in terms of training convergence.
  
  The larger the scale, the more it can leverage orthogonality to store and process diverse concepts efficiently without seeming redundancy in the sense of either linguistic syntax or semantics, enabling better disentangling of features which is crucial for capturing nuanced relationships in language. For the analogy task where "king - man + woman = queen", orthogonality ensures that
  the "gender" axis is distinct from the "royalty" axis and operations like subtraction and addition correspond to meaningful transformations in these orthogonal dimensions. Therefore orthogonality in high-dimensional spaces is one of the key reasons why scaling LLMs often leads to disproportionately better performance.
  
  
  ---
  
  