Answers.Body: <p>Nicola Bernini's answer is quite comprehensive. Here are my insights.</p>
  
  <p><strong>First of all, think whether you really need neural networks to solve your problem. Think whether traditional computer vision operations like edge detection/ region-based methods help you to solve your problem (OpenCV can help you here). Think about your data again. In case you decide to use neural networks, some things to try out:-</strong></p>
  
  <ol>
  <li><p>Your data set size is too small. Try to recall that we are learning to approximate functions (Universal approximation theorem). Less data + More parameters have high chance of overfitting. Use transfer learning. (Try Resizing the image and perform random resized square crop and use as input to your neural network. This may or may not work since I don't know what exactly you are doing). Also try data augmentations that make sense (i.e. : Vertical flip of traffic sign images don't work)</p></li>
  <li><p>Try reducing the learning rate/ using different learning rates for different parts of your network if you decide to use transfer learning.</p></li>
  <li><p>Check whether your train and test dataset distributions are the same. i.e.: Don't train with 95 % of label 0 and test with a set that has 95% label 1. I do not know whether your dataset is highly class imbalanced or whether you are doing some kind of anomaly detection.</p></li>
  <li><p>Think about optimizers. Try Adam if you haven't. </p></li>
  </ol>
  
  <p>If you need more help, try sharing the data and we can try to help you.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-24T16:13:31.687Z
Answers.LastActivityDate: 2019-08-24T16:13:31.687Z
Answers.UserDisplayName: keshik
Answers.Body: <h1>Overview</h1>
  
  <p>As it has already been observed, your main problem, beside the training related issues like fixing the learning rate, is you have basically <strong>no chance to learn such a big model woth such a small dataset ... from scratch</strong> </p>
  
  <p>So focusing on the real problem, here are some techniques you could use </p>
  
  <ul>
  <li>dataset augmentation </li>
  <li>transfer learning 
  
  <ul>
  <li>from a pretrained model </li>
  <li>from the encoder stage of an autoencoder (last resort option before getting into more advanced topics)</li>
  </ul></li>
  </ul>
  
  <h1>Dataset Augmentation</h1>
  
  <p>Add transformations to your dataset you want your classifier to learn to be invariant to </p>
  
  <p>Let's assume that </p>
  
  <ul>
  <li><p><span class="math-container">$I$</span> is an input image </p></li>
  <li><p><span class="math-container">$l$</span> its associated label </p></li>
  <li><p><span class="math-container">$f(\mathcal{I};\theta) \rightarrow \mathcal{I}$</span> is a parametric transformation that affects appearance but not semantic, for example it is a rotation of <span class="math-container">$\theta$</span> angle </p></li>
  </ul>
  
  <p>then you can augment your dataset by generating <span class="math-container">$\{I_{\theta}, l\}$</span> a set of transformed (e.g. rotated) images associated the same <span class="math-container">$l$</span> label </p>
  
  <h1>Transfer Learning</h1>
  
  <p>The fundamental idea of transfer learning is to re-use a NN which has been trained to solve a task, to solve other tasks retraining only a selected subset of the weights </p>
  
  <p>It means using a pre-trained convolutive backend, the part of the model with Conv2D and Pooling, and train dense layers with dropout only (but you should still probably think about reducing the dimensionality there) </p>
  
  <p>More formally think about representing your CNN Classifier as follows </p>
  
  <ul>
  <li><p><span class="math-container">$f_{C}(I; \theta_{X})$</span> : Convolutive Processing on Input Image </p>
  
  <ul>
  <li>it is the part of the CNN composed of <code>Conv2D</code> and <code>MaxPooling2D</code> layers </li>
  <li>the <span class="math-container">$\theta_{C}$</span> is the convolutive learnable weights set </li>
  </ul></li>
  <li><p><span class="math-container">$b = f_{C}(I; \theta_{C})$</span> : Bottleneck Feature Representation </p>
  
  <ul>
  <li>it is the result of <code>Flatten</code> layer </li>
  </ul></li>
  <li><p><span class="math-container">$f_{D}(b; \theta_{D})$</span> : Dense Processing </p>
  
  <ul>
  <li>it is the part of the model composed of <code>Dense</code> layers </li>
  <li>the <span class="math-container">$\theta_{D}$</span> is the dense learnable weights set </li>
  </ul></li>
  </ul>
  
  <p>The idea is to pick <span class="math-container">$\theta_{C}$</span> from a training performed on an another dataset, bigger than your current one, and keep it fixed while training in your task 
  This means reducing the number of parameters to be trained, however beware the dense layers account for most of the weights, as you can also see from your mode summary, which means you should also focus on reducing that number, for example reducing the bottleneck feature tensor size </p>
  
  <h2>Transfer Learning from Pre-Trained Model</h2>
  
  <p>For example, if your actual goal was to perform binary classification on some kind of MNIST-like data then you could use a convolutive backend from a CNN which has been pre-trained on the MNIST 0..9 classification task or you can train it yourself but is important is the <span class="math-container">$\theta_{C}$</span> weights will be learned from a MNIST dataset, which is much bigger than yours, even if the task is (slightly) different. </p>
  
  <p>Furthermore, in case of MNIST like data, please consider if you really need your full <code>80 x 130</code> resolution hence your input tensor, considering I can deduct from your model summary it is grayscale (no color), needs to be <span class="math-container">$(80,130,1)$</span> or you could rescale to the <code>28 x 28</code> MNIST resolution so you work with a smaller <span class="math-container">$(28,28,1)$</span> tensor </p>
  
  <p>My suggestion is to start from an architecture like this <a href="https://gist.github.com/mjbhobe/cf895231aab8fa45a2a23da1351d185d#file-mnist_keras_model-py" rel="noreferrer">MNIST Keras Model</a> as </p>
  
  <ul>
  <li>it has a bottleneck representation of 64 which could be enough for your task and </li>
  <li>also suggesting to remove the first dense layer so to significantly reduce <span class="math-container">$\theta_{D}$</span> the number of learnable paramters hence going for something like </li>
  </ul>
  
  
  
  <pre><code>  model = Sequential()
    # add Convolutional layers
    model.add(Conv2D(filters=32, kernel_size=(3,3), activation='relu', padding='same', input_shape=(10, 10, 1)))
    model.add(MaxPooling2D(pool_size=(2,2)))
    model.add(Conv2D(filters=64, kernel_size=(3,3), activation='relu', padding='same'))
    model.add(MaxPooling2D(pool_size=(2,2)))
    model.add(Conv2D(filters=64, kernel_size=(3,3), activation='relu', padding='same'))
    model.add(MaxPooling2D(pool_size=(2,2)))    
    model.add(Flatten())
    # output layer
    model.add(Dense(1, activation='sigmoid'))
  
  </code></pre>
  
  <p>then compile the model with <code>binary_crossentropy</code> loss and maybe start giving a try to <code>adam</code> optimizer </p>
  
  <h2>Transfer Learning from Autoencoder</h2>
  
  <p>If your data is so special you can't find any big enough and similar enough dataset to use this strategy and you do not come up with any transformation you could use 
  to perform dataset augmentation, without getting into advanced things, you could try to play one last card: use an Autoencoder to learn a compressed representation aimed at reconstructing the original image and perform transfer learning with the encoder only </p>
  
  <p>For example, again under the assumption of working with a <span class="math-container">$(28,28,1)$</span> tensor, you could start with an architecture like the following one </p>
  
  
  
  <pre><code>def build_ae(input_img): 
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)
    # (28,28,16)
  
    encoded = MaxPooling2D((8, 8), padding='same')(x)
    # (4,4,8)
  
  
    x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)
    # (4,4,8)
  
    x = UpSampling2D((8, 8))(x)
    # (16,16,8)
  
    x = Conv2D(16, (3, 3), activation='relu')(x)
    # Note: Convolving without padding='same' in order to get w-2 and h-2 dimensioality reduction so that following upsampling can lead to the desired 28x28 spatial resolution 
    # (14,14,8)
  
    x = UpSampling2D((2, 2))(x)
    # (28,28,8)
  
    decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)
    autoencoder = Model(input_img, decoded)
    return autoencoder
  
  </code></pre>
  
  <p>In this case, the full model has 2633 weights but the encoding stage consists only of Conv2D+Relu+MaxPooling which means in total <code>3x3x1x16</code> weights for the convolutive step and <code>16</code> weights for the relu for a total of 160 weights only and the latent representation is a <span class="math-container">$(4,4,8)$</span> tensor which means a 128 dimensional flattened tensor and hence assuming, as before, to perform the binary classification with a dense sigmoid layer it would mean 128+1 weights to learn in the actual binary classification task </p>
  
  <p>Of course it is possible to go for an even more compressed latent representation both on the spatial domain or channel domain with consequent reduced flattened vector dimensionality and ultimately even less weights to learn </p>
  
  <p>Would you share more details about your problem, also your dataset, we could try to help more </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-24T09:27:32.570Z
Answers.LastActivityDate: 2019-08-24T09:27:32.570Z
Answers.UserDisplayName: Nicola Bernini
Answers.Body: <p>Try lowering the learning rate.</p>
  
  <p>Such a loss curve can be indicative of a high learning rate. Due to a high learning rate the algorithm can take large steps in the direction of the gradient and miss the local minima. Then it will try to come back to the minima in the next step and overshoot it again. </p>
  
  <p>You may also try switching to a momentum-based GD algorithm. Such a training loss curve can be indicative of a loss contour like in <a href="http://ruder.io/optimizing-gradient-descent/index.html#momentum" rel="noreferrer">this example</a>, for which momentum-based GD methods are helpful.</p>
  
  <p>I noticed that you have a very small training set. You may have better luck with a larger training data (~1000 examples) or using a pre-trained Conv network as a starting point.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-27T05:18:53.303Z
Answers.Text: Lowering Learning rate did not help(0.1, 0.01, or 0.005 or even 0.001)
Answers.UserDisplayName: Krishnakumar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-21T16:41:58.217Z
Answers.LastActivityDate: 2019-08-21T16:41:58.217Z
Answers.UserDisplayName: Sabyasachi Ghosh
Answers.Body: <p>From my experience, this oscillation comes from:</p>
  <ul>
  <li>Too high learning rate: Weights change too quick.</li>
  <li>Too few neurons in layers: Not enough to fit.
  <ul>
  <li>When having not enough neurons, the network can't learn at all and the oscillation is due to failure to fit to global optimum; it's correct for some cases and wrong on other cases</li>
  </ul>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-26T06:55:56.960Z
Answers.LastActivityDate: 2021-11-24T02:34:07.927Z
Answers.LastEditDate: 2021-11-24T02:34:07.927Z
Answers.UserDisplayName: Dan D.
Body: <p><a href="https://i.sstatic.net/7FnY6.png" rel="noreferrer"><img src="https://i.sstatic.net/7FnY6.png" alt="enter image description here"></a></p>
  
  <p>I tried to create a simple model that receives an <span class="math-container">$80 \times 130$</span> pixel image. I only had 35 images and 10 test images. I trained this model for a binary classification task. The architecture of the model is described below.</p>
  
  <pre><code>conv2d_1 (Conv2D)            (None, 80, 130, 64)       640       
  _________________________________________________________________
  conv2d_2 (Conv2D)            (None, 78, 128, 64)       36928     
  _________________________________________________________________
  max_pooling2d_1 (MaxPooling2 (None, 39, 64, 64)        0         
  _________________________________________________________________
  dropout_1 (Dropout)          (None, 39, 64, 64)        0         
  _________________________________________________________________
  conv2d_3 (Conv2D)            (None, 39, 64, 128)       73856     
  _________________________________________________________________
  conv2d_4 (Conv2D)            (None, 37, 62, 128)       147584    
  _________________________________________________________________
  max_pooling2d_2 (MaxPooling2 (None, 18, 31, 128)       0         
  _________________________________________________________________
  dropout_2 (Dropout)          (None, 18, 31, 128)       0         
  _________________________________________________________________
  flatten_1 (Flatten)          (None, 71424)             0         
  _________________________________________________________________
  dense_1 (Dense)              (None, 512)               36569600  
  _________________________________________________________________
  dropout_3 (Dropout)          (None, 512)               0         
  _________________________________________________________________
  dense_2 (Dense)              (None, 1)                 513     
  </code></pre>
  
  <p>What could the oscillating training loss curve represent above? Why is the validation loss constant?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-21T13:39:13.577Z
Comments.Text: what loss function you using and this is ABSURDLY large for the amount of data you have. Your models gonna have real difficulty
Comments.UserDisplayName: mshlis
Tags: <training><computer-vision><deep-neural-networks><object-recognition>
Title: What could an oscillating training loss curve represent?
UserDisplayName: Krishnakumar
ThreadText: # What could an oscillating training loss curve represent?
  
  (Asked by: username_0 on 2019-08-21)
  
  I tried to create a simple model that receives an $80 \times 130$ pixel image. I only had 35 images and 10 test images. I trained this model for a binary classification task. The architecture of the model is described below.
  conv2d_1 (Conv2D)            (None, 80, 130, 64)       640       
  _________________________________________________________________
  conv2d_2 (Conv2D)            (None, 78, 128, 64)       36928     
  _________________________________________________________________
  max_pooling2d_1 (MaxPooling2 (None, 39, 64, 64)        0         
  _________________________________________________________________
  dropout_1 (Dropout)          (None, 39, 64, 64)        0         
  _________________________________________________________________
  conv2d_3 (Conv2D)            (None, 39, 64, 128)       73856     
  _________________________________________________________________
  conv2d_4 (Conv2D)            (None, 37, 62, 128)       147584    
  _________________________________________________________________
  max_pooling2d_2 (MaxPooling2 (None, 18, 31, 128)       0         
  _________________________________________________________________
  dropout_2 (Dropout)          (None, 18, 31, 128)       0         
  _________________________________________________________________
  flatten_1 (Flatten)          (None, 71424)             0         
  _________________________________________________________________
  dense_1 (Dense)              (None, 512)               36569600  
  _________________________________________________________________
  dropout_3 (Dropout)          (None, 512)               0         
  _________________________________________________________________
  dense_2 (Dense)              (None, 1)                 513     
  
  What could the oscillating training loss curve represent above? Why is the validation loss constant?
  
  **Tags:** training, computer-vision, deep-neural-networks, object-recognition
  > **Comments:**
  > **username_1**: what loss function you using and this is ABSURDLY large for the amount of data you have. Your models gonna have real difficulty  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_3 on 2019-08-24. Score: 8
  
  Overview
  As it has already been observed, your main problem, beside the training related issues like fixing the learning rate, is you have basically no chance to learn such a big model woth such a small dataset ... from scratch 
  So focusing on the real problem, here are some techniques you could use 
  
  dataset augmentation 
  transfer learning 
  
  
  from a pretrained model 
  from the encoder stage of an autoencoder (last resort option before getting into more advanced topics)
  
  
  Dataset Augmentation
  Add transformations to your dataset you want your classifier to learn to be invariant to 
  Let's assume that 
  
  $I$ is an input image 
  $l$ its associated label 
  $f(\mathcal{I};\theta) \rightarrow \mathcal{I}$ is a parametric transformation that affects appearance but not semantic, for example it is a rotation of $\theta$ angle 
  
  then you can augment your dataset by generating $\{I_{\theta}, l\}$ a set of transformed (e.g. rotated) images associated the same $l$ label 
  Transfer Learning
  The fundamental idea of transfer learning is to re-use a NN which has been trained to solve a task, to solve other tasks retraining only a selected subset of the weights 
  It means using a pre-trained convolutive backend, the part of the model with Conv2D and Pooling, and train dense layers with dropout only (but you should still probably think about reducing the dimensionality there) 
  More formally think about representing your CNN Classifier as follows 
  
  $f_{C}(I; \theta_{X})$ : Convolutive Processing on Input Image 
  
  it is the part of the CNN composed of Conv2D and MaxPooling2D layers 
  the $\theta_{C}$ is the convolutive learnable weights set 
  
  $b = f_{C}(I; \theta_{C})$ : Bottleneck Feature Representation 
  
  it is the result of Flatten layer 
  
  $f_{D}(b; \theta_{D})$ : Dense Processing 
  
  it is the part of the model composed of Dense layers 
  the $\theta_{D}$ is the dense learnable weights set 
  
  
  The idea is to pick $\theta_{C}$ from a training performed on an another dataset, bigger than your current one, and keep it fixed while training in your task 
  This means reducing the number of parameters to be trained, however beware the dense layers account for most of the weights, as you can also see from your mode summary, which means you should also focus on reducing that number, for example reducing the bottleneck feature tensor size 
  Transfer Learning from Pre-Trained Model
  For example, if your actual goal was to perform binary classification on some kind of MNIST-like data then you could use a convolutive backend from a CNN which has been pre-trained on the MNIST 0..9 classification task or you can train it yourself but is important is the $\theta_{C}$ weights will be learned from a MNIST dataset, which is much bigger than yours, even if the task is (slightly) different. 
  Furthermore, in case of MNIST like data, please consider if you really need your full 80 x 130 resolution hence your input tensor, considering I can deduct from your model summary it is grayscale (no color), needs to be $(80,130,1)$ or you could rescale to the 28 x 28 MNIST resolution so you work with a smaller $(28,28,1)$ tensor 
  My suggestion is to start from an architecture like this MNIST Keras Model as 
  
  it has a bottleneck representation of 64 which could be enough for your task and 
  also suggesting to remove the first dense layer so to significantly reduce $\theta_{D}$ the number of learnable paramters hence going for something like 
  
    model = Sequential()
    # add Convolutional layers
    model.add(Conv2D(filters=32, kernel_size=(3,3), activation='relu', padding='same', input_shape=(10, 10, 1)))
    model.add(MaxPooling2D(pool_size=(2,2)))
    model.add(Conv2D(filters=64, kernel_size=(3,3), activation='relu', padding='same'))
    model.add(MaxPooling2D(pool_size=(2,2)))
    model.add(Conv2D(filters=64, kernel_size=(3,3), activation='relu', padding='same'))
    model.add(MaxPooling2D(pool_size=(2,2)))    
    model.add(Flatten())
    # output layer
    model.add(Dense(1, activation='sigmoid'))
  
  
  then compile the model with binary_crossentropy loss and maybe start giving a try to adam optimizer 
  Transfer Learning from Autoencoder
  If your data is so special you can't find any big enough and similar enough dataset to use this strategy and you do not come up with any transformation you could use 
  to perform dataset augmentation, without getting into advanced things, you could try to play one last card: use an Autoencoder to learn a compressed representation aimed at reconstructing the original image and perform transfer learning with the encoder only 
  For example, again under the assumption of working with a $(28,28,1)$ tensor, you could start with an architecture like the following one 
  def build_ae(input_img): 
    x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)
    # (28,28,16)
  
    encoded = MaxPooling2D((8, 8), padding='same')(x)
    # (4,4,8)
  
  
    x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)
    # (4,4,8)
  
    x = UpSampling2D((8, 8))(x)
    # (16,16,8)
  
    x = Conv2D(16, (3, 3), activation='relu')(x)
    # Note: Convolving without padding='same' in order to get w-2 and h-2 dimensioality reduction so that following upsampling can lead to the desired 28x28 spatial resolution 
    # (14,14,8)
  
    x = UpSampling2D((2, 2))(x)
    # (28,28,8)
  
    decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)
    autoencoder = Model(input_img, decoded)
    return autoencoder
  
  
  In this case, the full model has 2633 weights but the encoding stage consists only of Conv2D+Relu+MaxPooling which means in total 3x3x1x16 weights for the convolutive step and 16 weights for the relu for a total of 160 weights only and the latent representation is a $(4,4,8)$ tensor which means a 128 dimensional flattened tensor and hence assuming, as before, to perform the binary classification with a dense sigmoid layer it would mean 128+1 weights to learn in the actual binary classification task 
  Of course it is possible to go for an even more compressed latent representation both on the spatial domain or channel domain with consequent reduced flattened vector dimensionality and ultimately even less weights to learn 
  Would you share more details about your problem, also your dataset, we could try to help more
  
  
  ---
  
  ### Answer by username_4 on 2019-08-21. Score: 7
  
  Try lowering the learning rate.
  
  Such a loss curve can be indicative of a high learning rate. Due to a high learning rate the algorithm can take large steps in the direction of the gradient and miss the local minima. Then it will try to come back to the minima in the next step and overshoot it again.
  
  You may also try switching to a momentum-based GD algorithm. Such a training loss curve can be indicative of a loss contour like in [this example](http://ruder.io/optimizing-gradient-descent/index.html#momentum), for which momentum-based GD methods are helpful.
  
  I noticed that you have a very small training set. You may have better luck with a larger training data (~1000 examples) or using a pre-trained Conv network as a starting point.
  
  > **Comments:**
  > **username_0**: Lowering Learning rate did not help(0.1, 0.01, or 0.005 or even 0.001)  
  
  ---
  
  ### Answer by username_2 on 2019-08-24. Score: 3
  
  Nicola Bernini's answer is quite comprehensive. Here are my insights.
  
  **First of all, think whether you really need neural networks to solve your problem. Think whether traditional computer vision operations like edge detection/ region-based methods help you to solve your problem (OpenCV can help you here). Think about your data again. In case you decide to use neural networks, some things to try out:-**
  
  1. Your data set size is too small. Try to recall that we are learning to approximate functions (Universal approximation theorem). Less data + More parameters have high chance of overfitting. Use transfer learning. (Try Resizing the image and perform random resized square crop and use as input to your neural network. This may or may not work since I don't know what exactly you are doing). Also try data augmentations that make sense (i.e. : Vertical flip of traffic sign images don't work)
  2. Try reducing the learning rate/ using different learning rates for different parts of your network if you decide to use transfer learning.
  3. Check whether your train and test dataset distributions are the same. i.e.: Don't train with 95 % of label 0 and test with a set that has 95% label 1. I do not know whether your dataset is highly class imbalanced or whether you are doing some kind of anomaly detection.
  4. Think about optimizers. Try Adam if you haven't.
  
  If you need more help, try sharing the data and we can try to help you.
  
  
  ---
  
  ### Answer by username_5 on 2019-08-26. Score: 1
  
  From my experience, this oscillation comes from:
  
  * Too high learning rate: Weights change too quick.
  * Too few neurons in layers: Not enough to fit.
    + When having not enough neurons, the network can't learn at all and the oscillation is due to failure to fit to global optimum; it's correct for some cases and wrong on other cases
  
  
  ---
  
  