Answers.Body: <p>Without seeing your code, I could be missing something. However, simple CNNs like the one you posted above, are notoriously difficult to train. It could be possible that you just haven't found the correct hyper parameters. Consider creating a script that runs through many different combinations of learning rates, l2 regularization terms, and dropouts. If that doesn't work, it might be the case that you just need more data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-24T07:40:57.360Z
Answers.Text: I have added code of model. When my model was bigger - it wasn't learning. I will try to experiment with hyper parameters, but I think it won't help. Everyone tells me, that I need more data. But how much data is good for my task? Thank you for your answer.
Answers.UserDisplayName: Mixadyt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-24T09:23:23.867Z
Answers.Text: Bigger CNNs should be accompanied with skip connections, which improve gradient flow and smooth the loss surface significantly. Also, generally, if you find yourself backed into a corner after you have rigorously tested your training pipeline, it helps to shift from a build mindset to and understanding one. What is the distribution of image in the train and test set? What are my convolutional layers learning? What does a 2d projection of the output of my final linear layer look like?
Answers.UserDisplayName: Jayson Meribe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-24T06:54:48.023Z
Answers.LastActivityDate: 2024-12-24T06:54:48.023Z
Answers.UserDisplayName: Jayson Meribe
Body: <p>I need to recognize 35 classes (9 numbers and 26 English letters). I have 700 images, <a href="https://i.sstatic.net/AJ3ZLfU8.png" rel="nofollow noreferrer">example</a>. I split it 80% - train, 20% - validation.
  My model is very small:</p>
  <pre><code>import torch
  from torch.nn import Module, Conv2d, MaxPool2d, ReLU, BatchNorm2d, BatchNorm1d, Linear, LeakyReLU, Softmax, AdaptiveMaxPool2d, Dropout
  
  class CNNBlock(Module):
      def __init__(self, in_channels, out_channels):
          super().__init__()
          self.conv = Conv2d(
              in_channels = in_channels,
              out_channels = out_channels,
              kernel_size = 3,
              padding = &quot;same&quot;
          )
          self.batchnorm = BatchNorm2d(num_features = out_channels)
          self.act = ReLU()
          self.dropout = Dropout(0.8)
          self.maxpool = MaxPool2d(
              kernel_size = 2,
              stride = 2
          )
  
      def forward(self, x):
          return self.maxpool(self.dropout(self.act(self.batchnorm(self.conv(x)))))
  
  class RecognitionModel(Module):
      def __init__(self):
          super().__init__()
          
          # Input 1x80x180
          self.block1 = CNNBlock(1, 32) # Block1 32x40x90
          self.block2 = CNNBlock(32, 64) # Block2 64x20x45
          self.gmp = AdaptiveMaxPool2d(output_size = 1) # GlobalMaxPool 64x1x1
          self.sqz = lambda x: x.squeeze() # Squeeze 64
          self.linear1 = Linear(
              in_features = 64,
              out_features = 32
          )
          self.batchnorm = BatchNorm1d(num_features = 32)
          self.act1 = LeakyReLU()
          self.dropout = Dropout(0.8)
          self.linear2 = Linear(
              in_features = 32,
              out_features = 35
          )
          self.act2 = Softmax(dim = 1)
  
      def forward(self, x):
          x = self.block1(x)
          x = self.block2(x)
          x = self.gmp(x)
          x = self.sqz(x)
          x = self.dropout(self.act1(self.batchnorm(self.linear1(x))))
          y = self.act2(self.linear2(x))
  
          return y
      
      def save(self):
          torch.save(self.state_dict(), './trainedmodel.pt')
  
      def load(self):
          self.load_state_dict(torch.load('./trainedmodel.pt', weights_only = True))
  </code></pre>
  <p>Also I have augmentations:</p>
  <pre><code>[
      Resize((80, 180)),
      RandomHorizontalFlip(),
      RandomRotation(20),
      RandomAffine(degrees=15, translate=(0.1, 0.1)),
      Grayscale(),
  
  ]
  </code></pre>
  <p>I also divide image by 255.</p>
  <p>I tried to set dropout probability to 0.5 on linear, 0.2 on conv:
  <a href="https://i.sstatic.net/wbVhBLY8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wbVhBLY8.png" alt="" /></a></p>
  <p>0.5 on linear, 0.5 on conv:</p>
  <p><a href="https://i.sstatic.net/O9boPN18.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/O9boPN18.png" alt="" /></a></p>
  <p>0.8 on linear, 0.8 on conv:</p>
  <p><a href="https://i.sstatic.net/82G65hjT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/82G65hjT.png" alt="" /></a></p>
  <p>But it always overfits. I'm using Adam(lr=0.001); CrossEntropyLoss. Why does it happen?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-24T06:37:19.597Z
Comments.Text: Can you post your model code?
Comments.UserDisplayName: Jayson Meribe
Tags: <machine-learning><convolutional-neural-networks><overfitting>
Title: What can I do with model that always overfits?
UserDisplayName: Mixadyt
ThreadText: # What can I do with model that always overfits?
  
  (Asked by: username_0 on 2024-12-23)
  
  I need to recognize 35 classes (9 numbers and 26 English letters). I have 700 images, [example](https://i.sstatic.net/AJ3ZLfU8.png). I split it 80% - train, 20% - validation.
  My model is very small:
  
  ```
  import torch
  from torch.nn import Module, Conv2d, MaxPool2d, ReLU, BatchNorm2d, BatchNorm1d, Linear, LeakyReLU, Softmax, AdaptiveMaxPool2d, Dropout
  
  class CNNBlock(Module):
      def __init__(self, in_channels, out_channels):
          super().__init__()
          self.conv = Conv2d(
              in_channels = in_channels,
              out_channels = out_channels,
              kernel_size = 3,
              padding = "same"
          )
          self.batchnorm = BatchNorm2d(num_features = out_channels)
          self.act = ReLU()
          self.dropout = Dropout(0.8)
          self.maxpool = MaxPool2d(
              kernel_size = 2,
              stride = 2
          )
  
      def forward(self, x):
          return self.maxpool(self.dropout(self.act(self.batchnorm(self.conv(x)))))
  
  class RecognitionModel(Module):
      def __init__(self):
          super().__init__()
          
          # Input 1x80x180
          self.block1 = CNNBlock(1, 32) # Block1 32x40x90
          self.block2 = CNNBlock(32, 64) # Block2 64x20x45
          self.gmp = AdaptiveMaxPool2d(output_size = 1) # GlobalMaxPool 64x1x1
          self.sqz = lambda x: x.squeeze() # Squeeze 64
          self.linear1 = Linear(
              in_features = 64,
              out_features = 32
          )
          self.batchnorm = BatchNorm1d(num_features = 32)
          self.act1 = LeakyReLU()
          self.dropout = Dropout(0.8)
          self.linear2 = Linear(
              in_features = 32,
              out_features = 35
          )
          self.act2 = Softmax(dim = 1)
  
      def forward(self, x):
          x = self.block1(x)
          x = self.block2(x)
          x = self.gmp(x)
          x = self.sqz(x)
          x = self.dropout(self.act1(self.batchnorm(self.linear1(x))))
          y = self.act2(self.linear2(x))
  
          return y
      
      def save(self):
          torch.save(self.state_dict(), './trainedmodel.pt')
  
      def load(self):
          self.load_state_dict(torch.load('./trainedmodel.pt', weights_only = True))
  
  ```
  
  Also I have augmentations:
  
  ```
  [
      Resize((80, 180)),
      RandomHorizontalFlip(),
      RandomRotation(20),
      RandomAffine(degrees=15, translate=(0.1, 0.1)),
      Grayscale(),
  
  ]
  
  ```
  
  I also divide image by 255.
  
  I tried to set dropout probability to 0.5 on linear, 0.2 on conv:
  [![](https://i.sstatic.net/wbVhBLY8.png)](https://i.sstatic.net/wbVhBLY8.png)
  
  0.5 on linear, 0.5 on conv:
  
  [![](https://i.sstatic.net/O9boPN18.png)](https://i.sstatic.net/O9boPN18.png)
  
  0.8 on linear, 0.8 on conv:
  
  [![](https://i.sstatic.net/82G65hjT.png)](https://i.sstatic.net/82G65hjT.png)
  
  But it always overfits. I'm using Adam(lr=0.001); CrossEntropyLoss. Why does it happen?
  
  **Tags:** machine-learning, convolutional-neural-networks, overfitting
  
  > **Comments:**
  > **username_1**: Can you post your model code?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-24. Score: 0
  
  Without seeing your code, I could be missing something. However, simple CNNs like the one you posted above, are notoriously difficult to train. It could be possible that you just haven't found the correct hyper parameters. Consider creating a script that runs through many different combinations of learning rates, l2 regularization terms, and dropouts. If that doesn't work, it might be the case that you just need more data.
  
  > **Comments:**
  > **username_0**: I have added code of model. When my model was bigger - it wasn't learning. I will try to experiment with hyper parameters, but I think it won't help. Everyone tells me, that I need more data. But how much data is good for my task? Thank you for your answer.  
  > **username_1**: Bigger CNNs should be accompanied with skip connections, which improve gradient flow and smooth the loss surface significantly. Also, generally, if you find yourself backed into a corner after you have rigorously tested your training pipeline, it helps to shift from a build mindset to and understanding one. What is the distribution of image in the train and test set? What are my convolutional layers learning? What does a 2d projection of the output of my final linear layer look like?  
  
  ---
  
  