Body: <p>I am training a classification neural network using Tensorflow2 (specifications below). The training goes well (good accuracy and no overfitting, apparently). During the training I monitor the learned kernels in the first convolutional layer. I'm expecting to see some patterns like edges or corners in the learned kernels, like <a href="https://cs231n.github.io/assets/cnn/weights.jpeg" rel="nofollow noreferrer">here</a>. However, my trained kernels look pretty random. Check the following examples after 30000 steps (1 step = 1 batch of 256 images):
  <a href="https://i.sstatic.net/29FJD.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/29FJD.png" alt="enter image description here" /></a></p>
  <p>Moreover, if I plot a specific kernel at different steps, the relation between different pixels does not seem to change much. Instead, it just seems do be scaled:
  <a href="https://i.sstatic.net/Mf1VJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Mf1VJ.png" alt="enter image description here" /></a></p>
  <p>Although the performance of the model seems good, this makes me think that there could be some overfitting in the training. What could be the explanation for this?</p>
  <p>Network specifications:</p>
  <ul>
  <li>2 classes (class 1: 200K samples, class 2: 64K samples)</li>
  <li>Adam optimizer</li>
  <li>Architecture like googlenet but with just 3 inception blocks</li>
  <li>L2 regularization (0.001) and dropout (0.2)</li>
  <li>Cross entropy loss, using sample weights to account for unbalanced dataset</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-08T14:38:51.873Z
Comments.Text: the size of your kernels seems pretty small, definitely hard to get something like in the linked example, it's like 5x5 vs 64x64. Also, have you tried to test your setting by saving kernels from a loaded pretrained model?
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-08T14:45:53.020Z
Comments.Text: I am using a custom architecture, so I cannot download a pre-trained model. Yes my kernels are 5x5, but shouldn't I expect some patterns like edges and corners for the first layer?
Comments.UserDisplayName: firion
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-08T15:50:37.913Z
Comments.Text: yes, but a kernel of 5x5 is really low resolution, so the question become, do the training image contain clear lines and corner clearly detectable at such resolution?
Comments.UserDisplayName: Edoardo Guerriero
Tags: <convolutional-neural-networks><tensorflow><filters><adam>
Title: Learned kernels in CNN seem just random patterns
UserDisplayName: firion
ThreadText: # Learned kernels in CNN seem just random patterns
  
  (Asked by: username_0 on 2022-07-08)
  
  I am training a classification neural network using Tensorflow2 (specifications below). The training goes well (good accuracy and no overfitting, apparently). During the training I monitor the learned kernels in the first convolutional layer. I'm expecting to see some patterns like edges or corners in the learned kernels, like [here](https://cs231n.github.io/assets/cnn/weights.jpeg). However, my trained kernels look pretty random. Check the following examples after 30000 steps (1 step = 1 batch of 256 images):
  [![enter image description here](https://i.sstatic.net/29FJD.png)](https://i.sstatic.net/29FJD.png)
  
  Moreover, if I plot a specific kernel at different steps, the relation between different pixels does not seem to change much. Instead, it just seems do be scaled:
  [![enter image description here](https://i.sstatic.net/Mf1VJ.png)](https://i.sstatic.net/Mf1VJ.png)
  
  Although the performance of the model seems good, this makes me think that there could be some overfitting in the training. What could be the explanation for this?
  
  Network specifications:
  
  * 2 classes (class 1: 200K samples, class 2: 64K samples)
  * Adam optimizer
  * Architecture like googlenet but with just 3 inception blocks
  * L2 regularization (0.001) and dropout (0.2)
  * Cross entropy loss, using sample weights to account for unbalanced dataset
  
  **Tags:** convolutional-neural-networks, tensorflow, filters, adam
  > **Comments:**
  > **username_1**: the size of your kernels seems pretty small, definitely hard to get something like in the linked example, it's like 5x5 vs 64x64. Also, have you tried to test your setting by saving kernels from a loaded pretrained model?  
  > **username_0**: I am using a custom architecture, so I cannot download a pre-trained model. Yes my kernels are 5x5, but shouldn't I expect some patterns like edges and corners for the first layer?  
  > **username_1**: yes, but a kernel of 5x5 is really low resolution, so the question become, do the training image contain clear lines and corner clearly detectable at such resolution?  
  
  ---
  
  ## 0 answers
  
  