Answers.Body: <p>It depends a lot on what type of architecture you are using. However, most of the standard architectures are quite stable and there is no need for much hypreparameter tuning.</p>
  <ol>
  <li>Choose whether you want to use a standard CNN architecture or a more advanced one like a <a href="https://pi-tau.github.io/posts/res-nets/" rel="nofollow noreferrer">ResNet</a>.</li>
  <li>Size? How many conv layers will you be using.</li>
  <li>Number of filters in each of the layers. This is where you might need a bit of tuning.</li>
  <li>You can try tuning the other parameters that are not specific to the problem, e.g. learning rate, regularization, dropout. If you already have a lot of feel for what works good and what not, then grid search according to your intuition would probably work pretty well. No need for random search in this case.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T10:33:49.370Z
Answers.LastActivityDate: 2023-09-05T10:33:49.370Z
Answers.UserDisplayName: pi-tau
Answers.Body: <p>You can play around with batch Size, epochs, learning rate.</p>
  <p>Start with a batch size of 8 and gradually increase two folds =&gt; 8,16,32</p>
  <p>Start with a higher learning rate to find the direction towards global minima, then reduce learning rate to avoid bouncing around and reach local minima</p>
  <p>Start with a low number of epochs to get quick feedback on the rest of hyperparameters. Once you have the optimal direction for hyperparameters move to higher number of epochs</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-08-10T19:48:03.403Z
Answers.LastActivityDate: 2022-08-10T19:48:03.403Z
Answers.UserDisplayName: Marib Sultan
Answers.Body: <p>Firstly when you say an object detection CNN, there are a huge number of model architectures available. Considering that you have narrowed down on your model architecture a CNN will have a few common layers like the ones below with hyperparameters you can tweak:</p>
  
  <ol>
  <li>Convolution Layer:- number of kernels, kernel size, stride length, padding</li>
  <li>MaxPooling Layer:- kernel size, stride length, padding</li>
  <li>Dense Layer:- size</li>
  <li>Dropout:- Percentage to keep/drop</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-23T14:15:14.170Z
Answers.Text: Thanks for your answer! Actually I do not intend to change the architecture, so the kernel size, the stride... are not concerned. The main goal of my question is HOW TO GO ABOUT THE TRAINING. What to strat from?
Answers.UserDisplayName: S.E.K.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-23T14:03:05.517Z
Answers.LastActivityDate: 2020-01-23T14:13:43.460Z
Answers.LastEditDate: 2020-01-23T14:13:43.460Z
Answers.UserDisplayName: keshav
Body: <p>I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read that doing a grid search for hyperparameters is not the best way to go about training and that random search is better in this case. Is random search really that good?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-15T15:03:38.637Z
Comments.Text: 1. Learning rate schedule (decrease rate, cyclic etc) 2.Momentum if used(including 0) 3. augmentation (random transformations values) if used
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-15T15:14:13.690Z
Comments.Text: @mirror2image momentum is fixed to 0.9 (it appears that it's common practice to fix it to this value if you want to focus on other hyperparameters). Augmentations are done, but I'm mostly talking about other hyperparameters : regularizarion, weight decay... AND MOST IMPORTANTLY: from where to start
Comments.UserDisplayName: S.E.K.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-15T15:18:19.397Z
Comments.Text: Weight decay is useless in most cases. There are a lot of different regularizations, use common sense for them. Fixed momentum is bad practice. Always good idea to check momentum 0, 0.6, 0.9 Start with different learning rate *schedules*
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-15T16:32:51.223Z
Comments.Text: @mirror2image so you say: I start with trying out different lr scheduling policies, and during each policy, I do a grid search of the learning rate,  momentum and the parameters of regulatization ?
Comments.UserDisplayName: S.E.K.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-15T17:55:44.500Z
Comments.Text: First - schedule and learning rate with grid search. After that momentum and regularization separately - it's highly likely they are  independent.
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-12T12:20:14.057Z
Comments.Text: This playlist by Andrew Ng helped me understand the different nuances of hyperparameter tuning. I'm sure it'll help you as well. https://www.youtube.com/watch?v=1waHlpKiNyY&list=PLkDaE6sCZn6Hn0vK8co82zjQtt3T2Nkqc&ab_channel=DeepLearningAI
Comments.UserDisplayName: Uday Sai
Tags: <convolutional-neural-networks><training><computer-vision><optimization><hyperparameter-optimization>
Title: When training a CNN, what are the hyperparameters to tune first?
UserDisplayName: S.E.K.
ThreadText: # When training a CNN, what are the hyperparameters to tune first?
  
  (Asked by: username_0 on 2020-01-15)
  
  I am training a convolutional neural network for object detection. Apart from the learning rate, what are the other hyperparameters that I should tune? And in what order of importance? Besides, I read that doing a grid search for hyperparameters is not the best way to go about training and that random search is better in this case. Is random search really that good?
  
  **Tags:** convolutional-neural-networks, training, computer-vision, optimization, hyperparameter-optimization
  
  > **Comments:**
  > **username_1**: 1. Learning rate schedule (decrease rate, cyclic etc) 2.Momentum if used(including 0) 3. augmentation (random transformations values) if used  
  > **username_0**: @username_1 momentum is fixed to 0.9 (it appears that it's common practice to fix it to this value if you want to focus on other hyperparameters). Augmentations are done, but I'm mostly talking about other hyperparameters : regularizarion, weight decay... AND MOST IMPORTANTLY: from where to start  
  > **username_1**: Weight decay is useless in most cases. There are a lot of different regularizations, use common sense for them. Fixed momentum is bad practice. Always good idea to check momentum 0, 0.6, 0.9 Start with different learning rate *schedules*  
  > **username_0**: @username_1 so you say: I start with trying out different lr scheduling policies, and during each policy, I do a grid search of the learning rate,  momentum and the parameters of regulatization ?  
  > **username_1**: First - schedule and learning rate with grid search. After that momentum and regularization separately - it's highly likely they are  independent.  
  > **username_2**: This playlist by Andrew Ng helped me understand the different nuances of hyperparameter tuning. I'm sure it'll help you as well. https://www.youtube.com/watch?v=1waHlpKiNyY&list=PLkDaE6sCZn6Hn0vK8co82zjQtt3T2Nkqc&ab_channel=DeepLearningAI  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2023-09-05. Score: 1
  
  It depends a lot on what type of architecture you are using. However, most of the standard architectures are quite stable and there is no need for much hypreparameter tuning.
  
  1. Choose whether you want to use a standard CNN architecture or a more advanced one like a [ResNet](https://username_3.github.io/posts/res-nets/).
  2. Size? How many conv layers will you be using.
  3. Number of filters in each of the layers. This is where you might need a bit of tuning.
  4. You can try tuning the other parameters that are not specific to the problem, e.g. learning rate, regularization, dropout. If you already have a lot of feel for what works good and what not, then grid search according to your intuition would probably work pretty well. No need for random search in this case.
  
  
  ---
  
  ### Answer by username_4 on 2022-08-10. Score: 0
  
  You can play around with batch Size, epochs, learning rate.
  
  Start with a batch size of 8 and gradually increase two folds => 8,16,32
  
  Start with a higher learning rate to find the direction towards global minima, then reduce learning rate to avoid bouncing around and reach local minima
  
  Start with a low number of epochs to get quick feedback on the rest of hyperparameters. Once you have the optimal direction for hyperparameters move to higher number of epochs
  
  
  ---
  
  ### Answer by username_5 on 2020-01-23. Score: 0
  
  Firstly when you say an object detection CNN, there are a huge number of model architectures available. Considering that you have narrowed down on your model architecture a CNN will have a few common layers like the ones below with hyperparameters you can tweak:
  
  1. Convolution Layer:- number of kernels, kernel size, stride length, padding
  2. MaxPooling Layer:- kernel size, stride length, padding
  3. Dense Layer:- size
  4. Dropout:- Percentage to keep/drop
  
  > **Comments:**
  > **username_0**: Thanks for your answer! Actually I do not intend to change the architecture, so the kernel size, the stride... are not concerned. The main goal of my question is HOW TO GO ABOUT THE TRAINING. What to strat from?  
  
  ---
  
  