Answers.Body: <p><strong>In theory</strong>, yes, using synthetic data generation. This involves applying transformations to the original images to generate new 'unique' images. Some standard techniques include rotating, flipping, stretching, zooming or brightening. Obviously not all of these make sense depending on the data. In your problem, zooming, stretching and brightening could be used but flipping should not. Rotation could work but only for small angles.</p>
  
  <p>Generally this is implemented by <em>replacing</em> the dataset for each epoch of training. Therefore, the number of images used in each training iteration is the same but the images themselves have been altered.</p>
  
  <p><strong>In practice</strong>, it's not a magic bullet. The reason a larger dataset generally yields better models is because the probability of a new feature falling within the feature distribution of the training data is higher. With synthetic data generation the new features are only marginally different to the original so even if the number of images to train on is increased, the feature distributions are not that different. There is a lot of variation in handwritten numbers so it would be very hard to guess how effective this would be without trying it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T15:44:24.700Z
Answers.LastActivityDate: 2020-04-03T15:44:24.700Z
Answers.UserDisplayName: tynowell
Answers.Body: <p>You can synthetically increase the number of samples. For example with augmentation or unsupervised adaption (<a href="https://arxiv.org/abs/1911.04252" rel="nofollow noreferrer">Self-training</a>). With augmentation you grant the system way more robustness so i would really recommend this. For example this <a href="https://github.com/aleju/imgaug" rel="nofollow noreferrer">github</a>. The problem with such small database sizes is that your test-set is also very small and you cannot test properly if your network generalizes well, or just overfits.</p>
  
  <p>You can try transfer learning with another larger network to adapt those feature extractors and use them on ur problem. That may work better than training a new one from scratch with so less labeled images. Hope i could help at least a little, stay tuned.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T15:48:17.113Z
Answers.Text: Thanks. I was wondering if there was a small tweak you could make to the back propagation algorithm to improve training.
Answers.UserDisplayName: david david
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T15:50:30.117Z
Answers.Text: Please if you find that instant contact me haha :D. Mhm maybe consider dropout, or adding noise on your system. Meaning add gaussian noise to the weigths itself, works quite good in self-learning and GAN´s, also stochastic depth is quite nice for the same kind of tweaking
Answers.UserDisplayName: Paul Higazi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T15:52:03.650Z
Answers.Text: i like the easy keras implementation. Maybe you want to try it here: https://keras.io/layers/noise/
Answers.UserDisplayName: Paul Higazi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T15:53:15.337Z
Answers.Text: I'm not a fan of python. I usually don't like using libraries either as they aren't very instructive but thanks a lot . I'll still check it out.
Answers.UserDisplayName: david david
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T15:46:34.263Z
Answers.LastActivityDate: 2020-04-03T15:46:34.263Z
Answers.UserDisplayName: Paul Higazi
Body: <p>I have created a neural network that is able to recognize images with the numbers 1-5. The issue is that I have a database of 16x5 images which ,unfortunately, is not proving enough as the neural network fails in the test set. Are there ways to improve a neural network's performance without using more data? The ANN has approximately a 90% accuracy on the training sets and a 50% accuracy in the test ones.</p>
  
  <p>Code:</p>
  
  <pre><code>clear
  graphics_toolkit("gnuplot")
  sigmoid = @(z) 1./(1 + exp(-z));
  sig_der = @(y) sigmoid(y).*(1-sigmoid(y));
  
  
  parse_image;   % This external f(x) loads the images so that they can be read. 
  %13x14
  num=0;
  for i=1:166
    if mod(i-1,10)&lt;=5 &amp;&amp; mod(i-1,10) &gt; 0
      num=num+1;
      data(:,num) = dlmread(strcat("/tmp/",num2str(i)))(:);
    end
  end
  
  
  
  function [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i)
    mid_layer(:,1)=sum(w1.*data(:,i));
    mid_layer(:,2)=sigmoid(mid_layer(:,1));
    last_layer(:,1)=sum(mid_layer(:,2).*w2);
    last_layer(:,2)=sigmoid(last_layer(:,1));
    exp_res=rem(i,5);
    if exp_res==0
      exp_res=5;
    end
    exp_result=zeros(5,1); exp_result(exp_res)=1;
    cost = exp_result-last_layer(:,2);
  end
  
  function [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i)
    delta(1:5) = cost;
    delta(6:20) = sum(cost' .* w2,2);
    w2 = w2 + 0.05 .* delta(1:5) .* mid_layer(:,2) .* sig_der(last_layer(:,1))';
    w1 = w1 + 0.05 .* delta(6:20) .* sig_der(mid_layer(:,1))' .* data(:,i);
  end
  
  w1=rand(182,15)./2.*(rand(182,15).*-2+1);
  w2=rand(15,5)./2.*(rand(15,5).*-2+1);
  
  for j=1:10000
    for i=[randperm(85)]
      [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i);
      [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i);
      cost_mem(j,i,:)=cost;
    end
  end
  </code></pre>
  
Tags: <neural-networks><backpropagation><optimization><matlab>
Title: How can I train a neural network if I don't have enough data?
UserDisplayName: david david
ThreadText: # How can I train a neural network if I don't have enough data?
  
  (Asked by: username_0 on 2020-04-03)
  
  I have created a neural network that is able to recognize images with the numbers 1-5. The issue is that I have a database of 16x5 images which ,unfortunately, is not proving enough as the neural network fails in the test set. Are there ways to improve a neural network's performance without using more data? The ANN has approximately a 90% accuracy on the training sets and a 50% accuracy in the test ones.
  Code:
  clear
  graphics_toolkit("gnuplot")
  sigmoid = @(z) 1./(1 + exp(-z));
  sig_der = @(y) sigmoid(y).*(1-sigmoid(y));
  
  
  parse_image;   % This external f(x) loads the images so that they can be read. 
  %13x14
  num=0;
  for i=1:166
    if mod(i-1,10)<=5 && mod(i-1,10) > 0
      num=num+1;
      data(:,num) = dlmread(strcat("/tmp/",num2str(i)))(:);
    end
  end
  
  
  
  function [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i)
    mid_layer(:,1)=sum(w1.*data(:,i));
    mid_layer(:,2)=sigmoid(mid_layer(:,1));
    last_layer(:,1)=sum(mid_layer(:,2).*w2);
    last_layer(:,2)=sigmoid(last_layer(:,1));
    exp_res=rem(i,5);
    if exp_res==0
      exp_res=5;
    end
    exp_result=zeros(5,1); exp_result(exp_res)=1;
    cost = exp_result-last_layer(:,2);
  end
  
  function [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i)
    delta(1:5) = cost;
    delta(6:20) = sum(cost' .* w2,2);
    w2 = w2 + 0.05 .* delta(1:5) .* mid_layer(:,2) .* sig_der(last_layer(:,1))';
    w1 = w1 + 0.05 .* delta(6:20) .* sig_der(mid_layer(:,1))' .* data(:,i);
  end
  
  w1=rand(182,15)./2.*(rand(182,15).*-2+1);
  w2=rand(15,5)./2.*(rand(15,5).*-2+1);
  
  for j=1:10000
    for i=[randperm(85)]
      [cost, mid_layer, last_layer] = forward(w1,w2,data,sigmoid,i);
      [w1, w2] = backprop(w1,w2,mid_layer,last_layer,data,cost,sig_der,sigmoid,i);
      cost_mem(j,i,:)=cost;
    end
  end
  
  **Tags:** neural-networks, backpropagation, optimization, matlab
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-04-03. Score: 3 | Accepted answer
  
  You can synthetically increase the number of samples. For example with augmentation or unsupervised adaption ([Self-training](https://arxiv.org/abs/1911.04252)). With augmentation you grant the system way more robustness so i would really recommend this. For example this [github](https://github.com/aleju/imgaug). The problem with such small database sizes is that your test-set is also very small and you cannot test properly if your network generalizes well, or just overfits.
  
  You can try transfer learning with another larger network to adapt those feature extractors and use them on ur problem. That may work better than training a new one from scratch with so less labeled images. Hope i could help at least a little, stay tuned.
  
  > **Comments:**
  > **username_0**: Thanks. I was wondering if there was a small tweak you could make to the back propagation algorithm to improve training.  
  > **username_2**: Please if you find that instant contact me haha :D. Mhm maybe consider dropout, or adding noise on your system. Meaning add gaussian noise to the weigths itself, works quite good in self-learning and GAN´s, also stochastic depth is quite nice for the same kind of tweaking  
  > **username_2**: i like the easy keras implementation. Maybe you want to try it here: https://keras.io/layers/noise/  
  > **username_0**: I'm not a fan of python. I usually don't like using libraries either as they aren't very instructive but thanks a lot . I'll still check it out.  
  
  ---
  
  ### Answer by username_1 on 2020-04-03. Score: 1
  
  **In theory**, yes, using synthetic data generation. This involves applying transformations to the original images to generate new 'unique' images. Some standard techniques include rotating, flipping, stretching, zooming or brightening. Obviously not all of these make sense depending on the data. In your problem, zooming, stretching and brightening could be used but flipping should not. Rotation could work but only for small angles.
  
  Generally this is implemented by *replacing* the dataset for each epoch of training. Therefore, the number of images used in each training iteration is the same but the images themselves have been altered.
  
  **In practice**, it's not a magic bullet. The reason a larger dataset generally yields better models is because the probability of a new feature falling within the feature distribution of the training data is higher. With synthetic data generation the new features are only marginally different to the original so even if the number of images to train on is increased, the feature distributions are not that different. There is a lot of variation in handwritten numbers so it would be very hard to guess how effective this would be without trying it.
  
  
  ---
  
  