Body: <p>I was reading the paper <a href="https://papers.nips.cc/paper/6114-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural-networks.pdf" rel="nofollow noreferrer">Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks</a> about improving the learning of an ANN using <em>weight normalization</em>.</p>
  
  <p>They consider standard artificial neural networks where the computation of each neuron consists in taking a weighted sum of input features, followed by an elementwise nonlinearity</p>
  
  <p><span class="math-container">$$y = \phi(\mathbf{x} \cdot \mathbf{w} + b)$$</span></p>
  
  <p>where <span class="math-container">$\mathbf{w}$</span> is a <span class="math-container">$k$</span>-dimensional weight vector, <span class="math-container">$b$</span> is a scalar bias term, <span class="math-container">$\mathbf{x}$</span> is a <span class="math-container">$k$</span>-dimensional vector of input features, <span class="math-container">$\phi(\cdot)$</span> denotes an elementwise nonlinearity and <span class="math-container">$y$</span> denotes the
  the scalar output of the neuron.</p>
  
  <p>They then propose to reparameterize each weight vector <span class="math-container">$\mathbf{w}$</span> in terms of a parameter vector <span class="math-container">$\mathbf{v}$</span> and a scalar parameter <span class="math-container">$g$</span> and to perform stochastic gradient descent with respect to those parameters instead.</p>
  
  <p><span class="math-container">$$
  \mathbf{w} = \frac{g}{\|\mathbf{v}\|} \mathbf{v}
  $$</span></p>
  
  <p>where <span class="math-container">$\mathbf{v}$</span> is a <span class="math-container">$k$</span>-dimensional vector, <span class="math-container">$g$</span> is a scalar, and <span class="math-container">$\|\mathbf{v}\|$</span> denotes the Euclidean norm of <span class="math-container">$\mathbf{v}$</span>. They call this  reparameterizaton <em>weight normalization</em>.</p>
  
  <p>What is this scalar <span class="math-container">$g$</span> used for, and where does it come from? Is <span class="math-container">$\mathbf{w}$</span> is the normalized weight? In general, how does weight normalization work? What is the intuition behind it?</p>
  
Tags: <neural-networks><deep-learning><papers><optimization><weight-normalization>
Title: How does weight normalization work?
UserDisplayName: Mike AI
ThreadText: # How does weight normalization work?
  
  (Asked by: username_0 on 2018-02-16)
  
  I was reading the paper [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://papers.nips.cc/paper/6114-weight-normalization-a-simple-reparameterization-to-accelerate-training-of-deep-neural-networks.pdf) about improving the learning of an ANN using *weight normalization*.
  
  They consider standard artificial neural networks where the computation of each neuron consists in taking a weighted sum of input features, followed by an elementwise nonlinearity
  
  $$y = \phi(\mathbf{x} \cdot \mathbf{w} + b)$$
  
  where $\mathbf{w}$ is a $k$-dimensional weight vector, $b$ is a scalar bias term, $\mathbf{x}$ is a $k$-dimensional vector of input features, $\phi(\cdot)$ denotes an elementwise nonlinearity and $y$ denotes the
  the scalar output of the neuron.
  
  They then propose to reparameterize each weight vector $\mathbf{w}$ in terms of a parameter vector $\mathbf{v}$ and a scalar parameter $g$ and to perform stochastic gradient descent with respect to those parameters instead.
  
  $$
  \mathbf{w} = \frac{g}{\|\mathbf{v}\|} \mathbf{v}
  $$
  
  where $\mathbf{v}$ is a $k$-dimensional vector, $g$ is a scalar, and $\|\mathbf{v}\|$ denotes the Euclidean norm of $\mathbf{v}$. They call this reparameterizaton *weight normalization*.
  
  What is this scalar $g$ used for, and where does it come from? Is $\mathbf{w}$ is the normalized weight? In general, how does weight normalization work? What is the intuition behind it?
  
  **Tags:** neural-networks, deep-learning, papers, optimization, weight-normalization
  
  ---
  
  ## 0 answers
  
  