Body: <p>In the context of a neural network <span class="math-container">$\hat{y} = f_\theta(\mathbf{x})$</span> with parameters <span class="math-container">$\theta$</span> that is trained to perform regression such that the prediction <span class="math-container">$\hat{\mathbf{y}} = [\hat{y}_1,\hat{y}_2,...,\hat{y}_N]$</span> is close the target <span class="math-container">$\mathbf{y} = [y_1,y_2,...,y_N]$</span>, the mean squared-error (MSE) loss function is:
  <span class="math-container">$$
  \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}) = \frac{1}{N} \sum_{i=1}^N (y_i - \hat{y}_i)^2
  $$</span>
  The parameters <span class="math-container">$\theta$</span> are then adjusted using the gradient descent update rule:
  <span class="math-container">$$
  \theta_{k+1} \leftarrow \theta_{k} - \alpha \cdot \nabla \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}_{\theta_k})
  $$</span>
  Where <span class="math-container">$\alpha$</span> is the learning rate. I am aware that if <span class="math-container">$\alpha$</span> is too small, the parameters <span class="math-container">$\theta$</span> might never converge, or is too slow to converge, to the optimal set of parameters <span class="math-container">$\theta^*$</span>, and if <span class="math-container">$\alpha$</span> is too large, the iterates of <span class="math-container">$\theta$</span> could oscillate and also never converge. My question has to do with the latter scenario, where <span class="math-container">$\alpha$</span> is too big, which leads to overshooting and oscillation.</p>
  <p>A good way of choosing <span class="math-container">$\alpha$</span> is using <a href="https://en.wikipedia.org/wiki/Backtracking_line_search" rel="nofollow noreferrer">backtracking line search</a>. However, because the neural network has many parameters, it is not practical to perform line search, and <span class="math-container">$\alpha$</span> needs to be chosen using another way.</p>
  <p>Is it possible to allow a larger value of the learning rate before overshooting and oscillation by &quot;elongating&quot; valleys in the MSE loss function <span class="math-container">$\mathcal{L}(\mathbf{y},\hat{\mathbf{y}})$</span>? More precisely, by transforming <span class="math-container">$\mathbf{y}$</span> and <span class="math-container">$\hat{\mathbf{y}}$</span> in some way before computing the loss? For example, in practice, I have found the following modification to the MSE loss function to be very helpful in avoiding overshooting and oscillation, even with large learning rates:
  <span class="math-container">$$
  \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}) = \frac{1}{N} \sum_{i=1}^N (\log(y_i + \epsilon) - \log(\hat{y}_i + \epsilon))^2
  $$</span>
  Where <span class="math-container">$\epsilon$</span> is a small value. However, I am not sure why this modification helps.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-17T19:13:42.497Z
Comments.Text: So... $\log^2\frac{y_i}{\hat{y}_i}$ ?
Comments.UserDisplayName: Kostya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-17T19:21:00.580Z
Comments.Text: Sure. This doesn't really explain why it helps to avoid overshooting though.
Comments.UserDisplayName: mhdadk
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-04-18T08:32:51.200Z
Comments.Text: The modified loss function possibly has much flatter gradient than the MSE loss, so the individual weight updates might be smaller, espcially if you are using an optimiser that has some kind of momentum.
Comments.UserDisplayName: Mike NZ
Tags: <neural-networks><objective-functions><math><gradient-descent><loss>
Title: Could the inputs of the mean squared-error loss function be transformed to allow larger learning rates?
UserDisplayName: mhdadk
ThreadText: # Could the inputs of the mean squared-error loss function be transformed to allow larger learning rates?
  
  (Asked by: username_0 on 2021-04-16)
  
  In the context of a neural network $\hat{y} = f\_\theta(\mathbf{x})$ with parameters $\theta$ that is trained to perform regression such that the prediction $\hat{\mathbf{y}} = [\hat{y}\_1,\hat{y}\_2,...,\hat{y}\_N]$ is close the target $\mathbf{y} = [y\_1,y\_2,...,y\_N]$, the mean squared-error (MSE) loss function is:
  $$
  \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}) = \frac{1}{N} \sum\_{i=1}^N (y\_i - \hat{y}\_i)^2
  $$
  The parameters $\theta$ are then adjusted using the gradient descent update rule:
  $$
  \theta\_{k+1} \leftarrow \theta\_{k} - \alpha \cdot \nabla \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}\_{\theta\_k})
  $$
  Where $\alpha$ is the learning rate. I am aware that if $\alpha$ is too small, the parameters $\theta$ might never converge, or is too slow to converge, to the optimal set of parameters $\theta^\*$, and if $\alpha$ is too large, the iterates of $\theta$ could oscillate and also never converge. My question has to do with the latter scenario, where $\alpha$ is too big, which leads to overshooting and oscillation.
  
  A good way of choosing $\alpha$ is using [backtracking line search](https://en.wikipedia.org/wiki/Backtracking_line_search). However, because the neural network has many parameters, it is not practical to perform line search, and $\alpha$ needs to be chosen using another way.
  
  Is it possible to allow a larger value of the learning rate before overshooting and oscillation by "elongating" valleys in the MSE loss function $\mathcal{L}(\mathbf{y},\hat{\mathbf{y}})$? More precisely, by transforming $\mathbf{y}$ and $\hat{\mathbf{y}}$ in some way before computing the loss? For example, in practice, I have found the following modification to the MSE loss function to be very helpful in avoiding overshooting and oscillation, even with large learning rates:
  $$
  \mathcal{L}(\mathbf{y},\hat{\mathbf{y}}) = \frac{1}{N} \sum\_{i=1}^N (\log(y\_i + \epsilon) - \log(\hat{y}\_i + \epsilon))^2
  $$
  Where $\epsilon$ is a small value. However, I am not sure why this modification helps.
  
  **Tags:** neural-networks, objective-functions, math, gradient-descent, loss
  > **Comments:**
  > **username_1**: So... $\log^2\frac{y_i}{\hat{y}_i}$ ?  
  > **username_0**: Sure. This doesn't really explain why it helps to avoid overshooting though.  
  > **username_2**: The modified loss function possibly has much flatter gradient than the MSE loss, so the individual weight updates might be smaller, espcially if you are using an optimiser that has some kind of momentum.  
  
  ---
  
  ## 0 answers
  
  