Answers.Body: <p>The issue doesn't come up terribly often. If you are only dealing with vectors, everything is either a row or column vector. It makes no difference which it is.</p>
  <p>A more relevant issue is whether one uses the so-called &quot;numerator layout&quot; or &quot;denominator layout&quot;. In the numerator layout, <span class="math-container">$\partial f / \partial x$</span> is <span class="math-container">$\mathbb{R}^{n\times m}$</span>, where <span class="math-container">$f \in \mathbb{R}^n$</span>, <span class="math-container">$x \in \mathbb{R}^m$</span>. The denominator layout is the other way (<span class="math-container">$\partial f / \partial x$</span> is <span class="math-container">$\mathbb{R}^{m \times n}$</span>).</p>
  <p>If you think of the gradient as a row vector, then you are using numerator layout.</p>
  <p>I think the numerator layout is overall more common. But there is no real convention, and if the paper content deals with Jacobian matrices it will definitely be mentioned somewhere in the text which layout they use.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-15T04:58:01.947Z
Answers.LastActivityDate: 2021-11-15T04:58:01.947Z
Answers.UserDisplayName: Taw
Body: <p>Consider the <a href="https://mml-book.github.io/book/mml-book.pdf#page=153" rel="nofollow noreferrer">following remark</a> about writing gradients from the chapter named <em>Vector Calculus</em> from the test book titled <strong><a href="https://mml-book.github.io/book/mml-book.pdf" rel="nofollow noreferrer">Mathematics for Machine Learning</a></strong> by <em>Marc Peter Deisenroth et al.</em></p>
  <blockquote>
  <p>Remark (Gradient as a Row Vector). It is not uncommon in the
  literature to define the gradient vector as a column vector, following
  the convention that vectors are generally column vectors. The reason
  why we define the gradient vector as a row vector is twofold: First,
  we can consistently generalize the gradient to vector-valued functions
  <span class="math-container">$f: \mathbb{R}^n \rightarrow \mathbb{R}^m$</span> (then the gradient becomes
  a matrix). Second, we can immediately apply the multi-variate chain
  rule without paying attention to the dimension of the gradient.</p>
  </blockquote>
  <p>The above remark implicitly says that there is no standard in writing the gradients. So, I can write a gradient of a scalar-valued multivariate function either as a column vector or as a row vector.</p>
  <p>But, I want to know which is more common in the AI community?</p>
  
Tags: <machine-learning><papers><notation><gradient><calculus>
Title: Which is more popular/common way of representing a gradient in AI community: as a row or column vector?
UserDisplayName: hanugm
ThreadText: # Which is more popular/common way of representing a gradient in AI community: as a row or column vector?
  
  (Asked by: username_0 on 2021-11-14)
  
  Consider the [following remark](https://mml-book.github.io/book/mml-book.pdf#page=153) about writing gradients from the chapter named *Vector Calculus* from the test book titled **[Mathematics for Machine Learning](https://mml-book.github.io/book/mml-book.pdf)** by *Marc Peter Deisenroth et al.*
  
  > Remark (Gradient as a Row Vector). It is not uncommon in the
  > literature to define the gradient vector as a column vector, following
  > the convention that vectors are generally column vectors. The reason
  > why we define the gradient vector as a row vector is twofold: First,
  > we can consistently generalize the gradient to vector-valued functions
  > $f: \mathbb{R}^n \rightarrow \mathbb{R}^m$ (then the gradient becomes
  > a matrix). Second, we can immediately apply the multi-variate chain
  > rule without paying attention to the dimension of the gradient.
  
  The above remark implicitly says that there is no standard in writing the gradients. So, I can write a gradient of a scalar-valued multivariate function either as a column vector or as a row vector.
  
  But, I want to know which is more common in the AI community?
  
  **Tags:** machine-learning, papers, notation, gradient, calculus
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-15. Score: 1 | Accepted answer
  
  The issue doesn't come up terribly often. If you are only dealing with vectors, everything is either a row or column vector. It makes no difference which it is.
  
  A more relevant issue is whether one uses the so-called "numerator layout" or "denominator layout". In the numerator layout, $\partial f / \partial x$ is $\mathbb{R}^{n\times m}$, where $f \in \mathbb{R}^n$, $x \in \mathbb{R}^m$. The denominator layout is the other way ($\partial f / \partial x$ is $\mathbb{R}^{m \times n}$).
  
  If you think of the gradient as a row vector, then you are using numerator layout.
  
  I think the numerator layout is overall more common. But there is no real convention, and if the paper content deals with Jacobian matrices it will definitely be mentioned somewhere in the text which layout they use.
  
  
  ---
  
  