Answers.Body: <p>There is no singular best set of hyperparameters. Even more, there is no real search algorithm for hyperparameters. You can do a grid search, but this obviously will take some time. Most people either do that or will try to handpick their parameters.</p>
  <p>A few other things to note: Initializing your weights at <code>[10^9,10^8]</code> doesn't seem right to me. They should be centered and close to zero initially. You should also take a look again how to set up train/test/val splits.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-03T09:40:21.410Z
Answers.Text: I meant the range $[10^{-9}, 10^{-8}]$. It was a typo. But almost all MNIST tutorials using TensorFlow seem to use the 6:1 train-test split ratio so I went for it.
Answers.UserDisplayName: Karthik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-03T09:47:30.750Z
Answers.Text: i was confused by the word set `chose the best set and trained the model using that set on the entire data set` but you are right, it is a good ratio. At the end of the day, good hyperparameters are most of the time default ones with actions taken to minimize overfitting.
Answers.UserDisplayName: N. Kiefer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-03T08:05:49.297Z
Answers.LastActivityDate: 2020-07-03T08:05:49.297Z
Answers.UserDisplayName: N. Kiefer
Body: <p>I wanted to understand back-propagation so I made a basic neural network library. I used momentum, with learning rate = <span class="math-container">$0.1$</span>, beta = <span class="math-container">$0.99$</span>, epochs = <span class="math-container">$200$</span>, batch size = <span class="math-container">$10$</span>, loss function is cross entropy and model structure is <span class="math-container">$784$</span>, <span class="math-container">$64$</span>, <span class="math-container">$64$</span>, <span class="math-container">$10$</span> and all layers use sigmoid. It performed terribly at first, so I initialized all the weights and biases in the range <span class="math-container">$[10^{-9}, 10^{-8}]$</span> and it worked. I am quite new to deep learning and I find TensorFlow doesn't seem as friendly to beginners who want to play around with hyper-parameters. How do you find the right hyper-parameters? I trained it on 100 digits (which took 10 minutes), tweaked hyper-parameters, chose the best set and trained the model using that set on the entire data set of <span class="math-container">$60,000$</span> images. I also found that halving the epochs and doubling the training set size gave better results. Are there fool proof heuristics to find good hyper-parameters? What is the best set of hyper-parameters (without regularization, dropout, etc) for MNIST digits? <a href="https://github.com/coderkarthiko/CustomNet" rel="nofollow noreferrer">Here</a> is the code for those who want to take a look.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-03T07:59:36.420Z
Comments.Text: Also, how is TensorFlow super fast? How does training in TensorFlow result in higher accuracy while training a custom network using same parameters take longer? I am not utilizing my GPU or anything so what gives?
Comments.UserDisplayName: Karthik
Tags: <deep-learning><hyperparameter-optimization>
Title: 96.91% accuracy on MNIST after 2 hours of training using custom made neural net library. Ways to improve?
UserDisplayName: Karthik
ThreadText: # 96.91% accuracy on MNIST after 2 hours of training using custom made neural net library. Ways to improve?
  
  (Asked by: username_0 on 2020-07-03)
  
  I wanted to understand back-propagation so I made a basic neural network library. I used momentum, with learning rate = $0.1$, beta = $0.99$, epochs = $200$, batch size = $10$, loss function is cross entropy and model structure is $784$, $64$, $64$, $10$ and all layers use sigmoid. It performed terribly at first, so I initialized all the weights and biases in the range $[10^{-9}, 10^{-8}]$ and it worked. I am quite new to deep learning and I find TensorFlow doesn't seem as friendly to beginners who want to play around with hyper-parameters. How do you find the right hyper-parameters? I trained it on 100 digits (which took 10 minutes), tweaked hyper-parameters, chose the best set and trained the model using that set on the entire data set of $60,000$ images. I also found that halving the epochs and doubling the training set size gave better results. Are there fool proof heuristics to find good hyper-parameters? What is the best set of hyper-parameters (without regularization, dropout, etc) for MNIST digits? [Here](https://github.com/coderkarthiko/CustomNet) is the code for those who want to take a look.
  
  **Tags:** deep-learning, hyperparameter-optimization
  
  > **Comments:**
  > **username_0**: Also, how is TensorFlow super fast? How does training in TensorFlow result in higher accuracy while training a custom network using same parameters take longer? I am not utilizing my GPU or anything so what gives?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-03. Score: 1
  
  There is no singular best set of hyperparameters. Even more, there is no real search algorithm for hyperparameters. You can do a grid search, but this obviously will take some time. Most people either do that or will try to handpick their parameters.
  
  A few other things to note: Initializing your weights at `[10^9,10^8]` doesn't seem right to me. They should be centered and close to zero initially. You should also take a look again how to set up train/test/val splits.
  
  > **Comments:**
  > **username_0**: I meant the range $[10^{-9}, 10^{-8}]$. It was a typo. But almost all MNIST tutorials using TensorFlow seem to use the 6:1 train-test split ratio so I went for it.  
  > **username_1**: i was confused by the word set `chose the best set and trained the model using that set on the entire data set` but you are right, it is a good ratio. At the end of the day, good hyperparameters are most of the time default ones with actions taken to minimize overfitting.  
  
  ---
  
  