Body: <p>I'm making a custom neural network framework (in C++, if that is of any help). When I train the model on MNIST, depending on how happy the network is feeling, it'll give me either 90%+ accuracy, or get stuck at 10-9% (on validation set).</p>
  <p>I shuffle all my data before feeding it to the neural net.</p>
  <p>Is there a better randomizer I should be using, or maybe I am not initializing my weights properly (Using <code>srand</code> to generate values between +/-0.1). Did I somehow hit a saddle point?</p>
  <p>My network consists of 784 size input layer, 256, 64, 32, 16 neuron hidden layers, all with RELU, and 10 output with SMAX</p>
  <p>Where should I start investigating based on this kind of behavior, when I can't even replicate what is going on?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-04T21:56:51.230Z
Comments.Text: This could be due to exploding gradients maybe. I'd suggest you to train with gradient clipping and see how the model learns.
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-04T21:58:35.443Z
Comments.Text: @SpiderRico wouldn't exploding gradients produce nan outputs (and also cause nan loss)? My loss and acc are both normal looking values, but I have not checked what the gradient looks like so I will look into it, thank you
Comments.UserDisplayName: Ilknur Mustafa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-04T21:59:41.827Z
Comments.Text: Correct, loss should be NaN but you mentioned accuracy getting stuck at 10%. This is equivalent to random guessing for MNIST (as there are 10 classes), and can be caused by exploding grads.
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-05T09:23:19.703Z
Comments.Text: How are you optimising your network? If you're using some form of gradient descent, is the learning rate you're using reasonable?
Comments.UserDisplayName: htl
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-03-05T11:41:46.227Z
Comments.Text: If you're using srand for a number between +-0.1, you should verify that none of your weights are initialized to zero.  If they are, they will stay there.
Comments.UserDisplayName: David Hoelzer
Tags: <neural-networks><accuracy><weights-initialization><mnist><validation>
Title: Why would my neural network have either an accuracy of 90% or 10% on the validation data, given a random initialization?
UserDisplayName: Ilknur Mustafa
ThreadText: # Why would my neural network have either an accuracy of 90% or 10% on the validation data, given a random initialization?
  
  (Asked by: username_0 on 2021-03-04)
  
  I'm making a custom neural network framework (in C++, if that is of any help). When I train the model on MNIST, depending on how happy the network is feeling, it'll give me either 90%+ accuracy, or get stuck at 10-9% (on validation set).
  
  I shuffle all my data before feeding it to the neural net.
  
  Is there a better randomizer I should be using, or maybe I am not initializing my weights properly (Using `srand` to generate values between +/-0.1). Did I somehow hit a saddle point?
  
  My network consists of 784 size input layer, 256, 64, 32, 16 neuron hidden layers, all with RELU, and 10 output with SMAX
  
  Where should I start investigating based on this kind of behavior, when I can't even replicate what is going on?
  
  **Tags:** neural-networks, accuracy, weights-initialization, mnist, validation
  
  > **Comments:**
  > **username_1**: This could be due to exploding gradients maybe. I'd suggest you to train with gradient clipping and see how the model learns.  
  > **username_0**: @username_1 wouldn't exploding gradients produce nan outputs (and also cause nan loss)? My loss and acc are both normal looking values, but I have not checked what the gradient looks like so I will look into it, thank you  
  > **username_1**: Correct, loss should be NaN but you mentioned accuracy getting stuck at 10%. This is equivalent to random guessing for MNIST (as there are 10 classes), and can be caused by exploding grads.  
  > **username_2**: How are you optimising your network? If you're using some form of gradient descent, is the learning rate you're using reasonable?  
  > **username_3**: If you're using srand for a number between +-0.1, you should verify that none of your weights are initialized to zero.  If they are, they will stay there.  
  
  ---
  
  ## 0 answers
  
  