Answers.Body: <p>You're describing exploding gradients, which as you noted is the opposite issue of vanishing gradients. Both these problems arise from the process of propagating and backpropagating activations/errors through the network. The intuition is that repeated multiplications of numbers less than 1 rapidly leads to a product of 0, while repeated multiplications of numbers greater than 1 rapidly leads to a product which tends to infinity.</p>
  <p>The number of matrix multiplications you need to perform increases with the number of layers, which is why this tends to be a problem with deeper networks. In particular, recurrent neural networks are infamously susceptible to vanishing/exploding gradients as a result of propagating through each layer multiple times corresponding to each time step.</p>
  <p>Careful choice of activation functions, proper weight initialization, and gradient clipping are the standard ways to address this problem, though more specialized solutions can arise depending on the context (for example, for RNNs, entirely new types of neurons, LSTMs and GRUs, were developed to address the issue).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-01T16:29:08.547Z
Answers.LastActivityDate: 2022-08-01T16:29:08.547Z
Answers.UserDisplayName: xojfqa
Body: <p>I'm trying to implement a simple neural network for classification (multi-class) as an exercise (written in C). During gradient descent, the weights and biases quickly get out of control and the gradient becomes infinite.</p>
  <p>I haven't been able to find any discussion of such problems (vanishing gradients is kind of the opposite).</p>
  <p>To be more specific, for testing I use a very simple network with 1 hidden layer and sigmoid as activation function. For the output layer I use softmax and logarithmic loss.</p>
  <p>The issue as I see it is that when an output activation is very small, for the derivative of the loss I basically get <code>1 / &lt;very_small number&gt;</code> and this leads to an enormous gradient.</p>
  <p>Am I doing something wrong in terms of network architecture?
  What is the typical way to deal with such problems?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-01T12:34:40.807Z
Comments.Text: have you tried to search "exploding gradients"? It's a well known issue along with vanishing gradients. Most of the time both problems comes from the weight initialization, what strategy are you using for that?
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-01T14:08:16.517Z
Comments.Text: @EdoardoGuerriero I'm using random values in `[-1, 1]`. I didn't know the term "exploding gradients", I'll check what info I can find, thanks.
Comments.UserDisplayName: martinkunev
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-08-01T15:00:29.827Z
Comments.Text: highly chances that your problem comes from the initialization. Usually the weights are positive defined, and you should go for values with mean of lower magnitude, like 0.05. Uniform or normal distributions are fine, but check also [xavier and kaiming initialization](https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79).
Comments.UserDisplayName: Edoardo Guerriero
Tags: <gradient-descent>
Title: Numerical problems with gradient descent
UserDisplayName: martinkunev
ThreadText: # Numerical problems with gradient descent
  
  (Asked by: username_0 on 2022-08-01)
  
  I'm trying to implement a simple neural network for classification (multi-class) as an exercise (written in C). During gradient descent, the weights and biases quickly get out of control and the gradient becomes infinite.
  
  I haven't been able to find any discussion of such problems (vanishing gradients is kind of the opposite).
  
  To be more specific, for testing I use a very simple network with 1 hidden layer and sigmoid as activation function. For the output layer I use softmax and logarithmic loss.
  
  The issue as I see it is that when an output activation is very small, for the derivative of the loss I basically get `1 /`  and this leads to an enormous gradient.
  
  Am I doing something wrong in terms of network architecture?
  What is the typical way to deal with such problems?
  
  **Tags:** gradient-descent
  > **Comments:**
  > **username_1**: have you tried to search "exploding gradients"? It's a well known issue along with vanishing gradients. Most of the time both problems comes from the weight initialization, what strategy are you using for that?  
  > **username_0**: @EdoardoGuerriero I'm using random values in `[-1, 1]`. I didn't know the term "exploding gradients", I'll check what info I can find, thanks.  
  > **username_1**: highly chances that your problem comes from the initialization. Usually the weights are positive defined, and you should go for values with mean of lower magnitude, like 0.05. Uniform or normal distributions are fine, but check also [xavier and kaiming initialization](https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-08-01. Score: 3
  
  You're describing exploding gradients, which as you noted is the opposite issue of vanishing gradients. Both these problems arise from the process of propagating and backpropagating activations/errors through the network. The intuition is that repeated multiplications of numbers less than 1 rapidly leads to a product of 0, while repeated multiplications of numbers greater than 1 rapidly leads to a product which tends to infinity.
  
  The number of matrix multiplications you need to perform increases with the number of layers, which is why this tends to be a problem with deeper networks. In particular, recurrent neural networks are infamously susceptible to vanishing/exploding gradients as a result of propagating through each layer multiple times corresponding to each time step.
  
  Careful choice of activation functions, proper weight initialization, and gradient clipping are the standard ways to address this problem, though more specialized solutions can arise depending on the context (for example, for RNNs, entirely new types of neurons, LSTMs and GRUs, were developed to address the issue).
  
  
  ---
  
  