Answers.Body: <p>Looking at the paper, it seems to me that they are not using orthogonal in a literal, mathematics (or geometric) sense.  Instead, I read that as two things (especially since the word &quot;ablation&quot; appears later in the sentence):</p>
  <ul>
  <li>They are attempting to use lots of fancy words</li>
  <li>They are simply indicating that these changes are separate from and have no impact on (at least they claim this to be so) the &quot;experimental factors [they] consider in [their] empirical survey...&quot;</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-14T22:06:42.837Z
Answers.LastActivityDate: 2021-09-15T11:57:31.593Z
Answers.LastEditDate: 2021-09-15T11:57:31.593Z
Answers.UserDisplayName: David Hoelzer
Answers.Body: <p>&quot;Orthogonal&quot; is often used to mean &quot;independent&quot;, as in &quot;independent variable which does not correlate with the other variables&quot;. I believe this terminology originates from <a href="https://en.wikipedia.org/wiki/Principal_component_analysis" rel="nofollow noreferrer">principal component analysis</a>, where uncorrelated variation would be along orthogonal axes.</p>
  <p>Or, in the words of the <a href="https://en.wikipedia.org/wiki/Orthogonality" rel="nofollow noreferrer">Wikipedia article on orthogonality</a> applied to computer science:</p>
  <blockquote>
  <p>Orthogonality is a system design property which guarantees that modifying the technical effect produced by a component of a system neither creates nor propagates side effects to other components of the system.</p>
  </blockquote>
  <p>So in this excerpt they state that their changes do not affect anything else (because they are independent/uncorrelated), and can hence be discussed elsewhere.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-15T10:14:40.903Z
Answers.LastActivityDate: 2021-09-15T10:14:40.903Z
Answers.UserDisplayName: Oliver Mason
Body: <p>Here's a quote from the <code>T5 paper</code> (T5 stands for &quot;Text-to-Text Transfer Transformer&quot;) titled <a href="https://www.jmlr.org/papers/volume21/20-074/20-074.pdf" rel="nofollow noreferrer">Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer</a> by <em>Colin Raffel et al.</em>:</p>
  <blockquote>
  <p>To summarize, our model is roughly equivalent to the original
  Transformer proposed by Vaswani et al. (2017) with the exception of
  removing the Layer Norm bias, placing the layer normalization outside
  the residual path, and using a different position embedding scheme.
  Since these architectural changes are <strong>orthogonal</strong> to the experimental
  factors we consider in our empirical survey of transfer learning, we
  leave the ablation of their impact for future work.</p>
  </blockquote>
  <p>What exactly does 'orthogonal' mean in this context? Also, is it just me or have I seen the word used in a similar way before, but can't remember where?</p>
  
Tags: <machine-learning><natural-language-processing><papers><transformer>
Title: Why do the authors of the T5 paper say that the "architectural changes are orthogonal to the experimental factors"?
UserDisplayName: asymmetryFan
ThreadText: # Why do the authors of the T5 paper say that the "architectural changes are orthogonal to the experimental factors"?
  
  (Asked by: username_0 on 2021-09-14)
  
  Here's a quote from the `T5 paper` (T5 stands for "Text-to-Text Transfer Transformer") titled [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://www.jmlr.org/papers/volume21/20-074/20-074.pdf) by *Colin Raffel et al.*:
  
  > To summarize, our model is roughly equivalent to the original
  > Transformer proposed by Vaswani et al. (2017) with the exception of
  > removing the Layer Norm bias, placing the layer normalization outside
  > the residual path, and using a different position embedding scheme.
  > Since these architectural changes are **orthogonal** to the experimental
  > factors we consider in our empirical survey of transfer learning, we
  > leave the ablation of their impact for future work.
  
  What exactly does 'orthogonal' mean in this context? Also, is it just me or have I seen the word used in a similar way before, but can't remember where?
  
  **Tags:** machine-learning, natural-language-processing, papers, transformer
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-09-15. Score: 3 | Accepted answer
  
  "Orthogonal" is often used to mean "independent", as in "independent variable which does not correlate with the other variables". I believe this terminology originates from [principal component analysis](https://en.wikipedia.org/wiki/Principal_component_analysis), where uncorrelated variation would be along orthogonal axes.
  
  Or, in the words of the [Wikipedia article on orthogonality](https://en.wikipedia.org/wiki/Orthogonality) applied to computer science:
  
  > Orthogonality is a system design property which guarantees that modifying the technical effect produced by a component of a system neither creates nor propagates side effects to other components of the system.
  
  So in this excerpt they state that their changes do not affect anything else (because they are independent/uncorrelated), and can hence be discussed elsewhere.
  
  
  ---
  
  ### Answer by username_1 on 2021-09-14. Score: 2
  
  Looking at the paper, it seems to me that they are not using orthogonal in a literal, mathematics (or geometric) sense. Instead, I read that as two things (especially since the word "ablation" appears later in the sentence):
  
  * They are attempting to use lots of fancy words
  * They are simply indicating that these changes are separate from and have no impact on (at least they claim this to be so) the "experimental factors [they] consider in [their] empirical survey..."
  
  
  ---
  
  