Answers.Body: <p>Since you already tried to gather more data along with data augmentation, and dropout. I can suggest you to use:</p>
  <ul>
  <li><strong>Weight decay</strong>, but in the form of <a href="https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/AdamW" rel="nofollow noreferrer">AdamW</a> (or similar for SGD) in which the &quot;W&quot; stands for decoupled weight decay update. In this way the magnitude of the penalty is both easier to control and interpret. I think you should use a coefficient of 1e-4 or larger.</li>
  <li><strong>Label smoothing</strong>, with an <span class="math-container">$\epsilon=0.05$</span> or <span class="math-container">$0.01$</span>.</li>
  <li><strong>Stochastic depth</strong>: is a regularization technique specific for networks with <em>skip connections</em>, basically dropping a residual path with probability <span class="math-container">$p$</span> (think of it as a dropout over entire skip connections.)</li>
  <li><strong>Advanced image augmentations</strong>: like training on <em>random crops</em> of your images (to increase their number; also be sure to not crop much otherwise the content will be changed), <em>cutout</em> (zero random portion of image), <a href="https://arxiv.org/abs/1710.09412" rel="nofollow noreferrer">mixup</a> (blend to random images and their classes as well), and <a href="https://arxiv.org/abs/1905.04899" rel="nofollow noreferrer">cutmix</a> (as before with also cutout.)</li>
  </ul>
  <p>If the issue still persists, try to pre-train on a different dataset and/or use a different network architecture.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-25T13:05:45.513Z
Answers.Text: Thanks for answering. I'm currently using a sgd optimizer with a decay rate of 1e-6. 
  I've just added the stoch dept per each conv2D layer with linear decay surviv rate, up to 50%. I reached a 80% val_accuracy before early stopping.
  I suppose pretraining can't be adopted to my personal use case, so I'll be exploring some other architectures, hoping for better results.
Answers.UserDisplayName: elbarto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-25T15:48:22.223Z
Answers.Text: That's useful information, you should add that to your question. Well, if you target dataset is not covered by ImageNet than it won't help much.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-24T20:54:13.727Z
Answers.LastActivityDate: 2023-05-25T15:46:01.630Z
Answers.LastEditDate: 2023-05-25T15:46:01.630Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am trying to train a MobileNetV2 on a custom dataset, to image Classification task.
  Cardinality is 864 images, split in 70%/20%/10%, balanced between the 3 different classes.</p>
  <p>Weights are pre-loaded from imagenet, I froze the net and I added to the bottom of the net a GlobalAveragePooling, a Dropout (with 50% drop probability), and a Dense layer with 3 classes and softmax as activation function, since i want the output layer to give me an output like (1,0,0) if the inference image is from the first class, and so on.</p>
  <ul>
  <li>image size: 96x96 (I normalized, too)</li>
  <li>batch_size: 32</li>
  <li>Learning rate: 0.001</li>
  <li>trainable params: 3843</li>
  <li>optimizer: sgd ('adam' doesn't improve my accuracy)</li>
  <li>loss: categorical cross entropy</li>
  <li>metrics: accuracy</li>
  </ul>
  <p>Training for 20 epochs gives me these results:
  <a href="https://i.sstatic.net/Sh1Cr.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Sh1Cr.png" alt="First learning curve" /></a></p>
  <p>After that I decided to try some fine-tuning, by freezing only the first 100 layers of the net.
  Trained again for 10 epochs, that's what I get:</p>
  <p><a href="https://i.sstatic.net/5z2AB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5z2AB.png" alt="Learning Curve after fine-tuning" /></a></p>
  <p>My net is overfitting, but I don't know why it's happening and what am I expected to do in order to improve my accuracy.</p>
  <p>Edit: I also tried increasing dataset images with some source images or even with some data augmentation, up to more than 3K images, but it didn't work out at any rate.</p>
  <p>2nd edit:  I'm currently using a sgd optimizer with a decay rate of 1e-6. I've just added the stoch dept per each conv2D layer with linear decay surviv rate, up to 50%. I reached a 80% val_accuracy before early stopping. I suppose pretraining can't be adopted to my personal use case, so I'll be exploring some other architectures, hoping for better results.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-16T10:01:14.320Z
Comments.Text: Hello. Welcome to this community! Could you please put your **specific question** in the title? "MobileNet not improving" is not really a question although it describes a situation.
Comments.UserDisplayName: nbro
Tags: <machine-learning><deep-learning><overfitting><mobile-net-v2>
Title: MobileNet validation loss not decreasing over time
UserDisplayName: elbarto
ThreadText: # MobileNet validation loss not decreasing over time
  
  (Asked by: username_0 on 2023-05-16)
  
  I am trying to train a MobileNetV2 on a custom dataset, to image Classification task.
  Cardinality is 864 images, split in 70%/20%/10%, balanced between the 3 different classes.
  
  Weights are pre-loaded from imagenet, I froze the net and I added to the bottom of the net a GlobalAveragePooling, a Dropout (with 50% drop probability), and a Dense layer with 3 classes and softmax as activation function, since i want the output layer to give me an output like (1,0,0) if the inference image is from the first class, and so on.
  
  * image size: 96x96 (I normalized, too)
  * batch\_size: 32
  * Learning rate: 0.001
  * trainable params: 3843
  * optimizer: sgd ('adam' doesn't improve my accuracy)
  * loss: categorical cross entropy
  * metrics: accuracy
  
  Training for 20 epochs gives me these results:
  [![First learning curve](https://i.sstatic.net/Sh1Cr.png)](https://i.sstatic.net/Sh1Cr.png)
  
  After that I decided to try some fine-tuning, by freezing only the first 100 layers of the net.
  Trained again for 10 epochs, that's what I get:
  
  [![Learning Curve after fine-tuning](https://i.sstatic.net/5z2AB.png)](https://i.sstatic.net/5z2AB.png)
  
  My net is overfitting, but I don't know why it's happening and what am I expected to do in order to improve my accuracy.
  
  Edit: I also tried increasing dataset images with some source images or even with some data augmentation, up to more than 3K images, but it didn't work out at any rate.
  
  2nd edit: I'm currently using a sgd optimizer with a decay rate of 1e-6. I've just added the stoch dept per each conv2D layer with linear decay surviv rate, up to 50%. I reached a 80% val\_accuracy before early stopping. I suppose pretraining can't be adopted to my personal use case, so I'll be exploring some other architectures, hoping for better results.
  
  **Tags:** machine-learning, deep-learning, overfitting, mobile-net-v2
  > **Comments:**
  > **username_1**: Hello. Welcome to this community! Could you please put your **specific question** in the title? "MobileNet not improving" is not really a question although it describes a situation.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-05-24. Score: 0
  
  Since you already tried to gather more data along with data augmentation, and dropout. I can suggest you to use:
  
  * **Weight decay**, but in the form of [AdamW](https://www.tensorflow.org/addons/api_docs/python/tfa/optimizers/AdamW) (or similar for SGD) in which the "W" stands for decoupled weight decay update. In this way the magnitude of the penalty is both easier to control and interpret. I think you should use a coefficient of 1e-4 or larger.
  * **Label smoothing**, with an $\epsilon=0.05$ or $0.01$.
  * **Stochastic depth**: is a regularization technique specific for networks with *skip connections*, basically dropping a residual path with probability $p$ (think of it as a dropout over entire skip connections.)
  * **Advanced image augmentations**: like training on *random crops* of your images (to increase their number; also be sure to not crop much otherwise the content will be changed), *cutout* (zero random portion of image), [mixup](https://arxiv.org/abs/1710.09412) (blend to random images and their classes as well), and [cutmix](https://arxiv.org/abs/1905.04899) (as before with also cutout.)
  
  If the issue still persists, try to pre-train on a different dataset and/or use a different network architecture.
  
  > **Comments:**
  > **username_0**: Thanks for answering. I'm currently using a sgd optimizer with a decay rate of 1e-6.  I've just added the stoch dept per each conv2D layer with linear decay surviv rate, up to 50%. I reached a 80% val_accuracy before early stopping. I suppose pretraining can't be adopted to my personal use case, so I'll be exploring some other architectures, hoping for better results.  
  > **username_2**: That's useful information, you should add that to your question. Well, if you target dataset is not covered by ImageNet than it won't help much.  
  
  ---
  
  