<ul>
<li>It is common during the training of <strong>Neural Networks</strong> for <strong>accuracy to improve for a while</strong> and <strong>then get worse</strong> -- in general, This is caused by <strong>over-fitting.</strong> It's also fairly common for the Neural Network to <strong>&quot;get UNLUCKY</strong> and get knocked into a <strong>BAD sectors of parameter space</strong> corresponding to a sudden decrease in accuracy -- sometimes it can recover from this quickly, but sometimes not.</li>
</ul>
<p><a href="https://i.sstatic.net/cFiA1.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/cFiA1.jpg" alt="image" /></a></p>
<ul>
<li><p>In general, lowering your <strong>learning rate</strong> is a <strong>good approach</strong> to this kind of problem. Also, setting a <strong>learning rate schedule</strong> like FactorScheduler can help you achieve <strong>more stable convergence</strong> by <strong>lowering the learning rate every few epochs</strong>. In fact, this can sometimes cover up mistakes in picking an initial learning rate that is too high.</p>
</li>
<li><p>you can try using mini-batches.</p>
</li>
<li><p>The error (Entropy) with log functions must be used <strong>precisely</strong>.</p>
</li>
</ul>
<hr />
<ul>
<li><p><a href="https://www.quora.com/What-are-some-possible-reasons-why-training-accuracy-would-decrease-over-time" rel="nofollow noreferrer">What are some possible reasons why training accuracy would decrease over time?</a></p>
</li>
<li><p>Check out in this <a href="https://arxiv.org/abs/1506.01186" rel="nofollow noreferrer">paper</a> a novel way to change the learning rate cyclically.</p>
</li>
</ul>

CC BY-SA 4.0
2020-05-05T10:22:28.413Z
2021-09-26T08:38:03.340Z
2021-09-26T08:38:03.340Z
Faizy
<p>I was following a tutorial about Feed-Forward Networks and wrote this code for a simple FFN :</p>
<pre><code>class FirstFFNetwork:
  
  #intialize the parameters
  def __init__(self):
    self.w1 = np.random.randn()
    self.w2 = np.random.randn()
    self.w3 = np.random.randn()
    self.w4 = np.random.randn()
    self.w5 = np.random.randn()
    self.w6 = np.random.randn()
    self.b1 = 0
    self.b2 = 0
    self.b3 = 0
  
  def sigmoid(self, x):
    return 1.0/(1.0 + np.exp(-x))
  
  def forward_pass(self, x):
    #forward pass - preactivation and activation
    self.x1, self.x2 = x
    self.a1 = self.w1*self.x1 + self.w2*self.x2 + self.b1
    self.h1 = self.sigmoid(self.a1)
    self.a2 = self.w3*self.x1 + self.w4*self.x2 + self.b2
    self.h2 = self.sigmoid(self.a2)
    self.a3 = self.w5*self.h1 + self.w6*self.h2 + self.b3
    self.h3 = self.sigmoid(self.a3)
    return self.h3
  
  def grad(self, x, y):
    #back propagation
    self.forward_pass(x)
    
    self.dw5 = (self.h3-y) * self.h3*(1-self.h3) * self.h1
    self.dw6 = (self.h3-y) * self.h3*(1-self.h3) * self.h2
    self.db3 = (self.h3-y) * self.h3*(1-self.h3)
    
    self.dw1 = (self.h3-y) * self.h3*(1-self.h3) * self.w5 * self.h1*(1-self.h1) * self.x1
    self.dw2 = (self.h3-y) * self.h3*(1-self.h3) * self.w5 * self.h1*(1-self.h1) * self.x2
    self.db1 = (self.h3-y) * self.h3*(1-self.h3) * self.w5 * self.h1*(1-self.h1)
  
    self.dw3 = (self.h3-y) * self.h3*(1-self.h3) * self.w6 * self.h2*(1-self.h2) * self.x1
    self.dw4 = (self.h3-y) * self.h3*(1-self.h3) * self.w6 * self.h2*(1-self.h2) * self.x2
    self.db2 = (self.h3-y) * self.h3*(1-self.h3) * self.w6 * self.h2*(1-self.h2)
    
  
  def fit(self, X, Y, epochs=1, learning_rate=1, initialise=True, display_loss=False):
    
    # initialise w, b
    if initialise:
      self.w1 = np.random.randn()
      self.w2 = np.random.randn()
      self.w3 = np.random.randn()
      self.w4 = np.random.randn()
      self.w5 = np.random.randn()
      self.w6 = np.random.randn()
      self.b1 = 0
      self.b2 = 0
      self.b3 = 0
      
    if display_loss:
      loss = {}
    
    for i in tqdm_notebook(range(epochs), total=epochs, unit=&quot;epoch&quot;):
      dw1, dw2, dw3, dw4, dw5, dw6, db1, db2, db3 = [0]*9
      for x, y in zip(X, Y):
        self.grad(x, y)
        dw1 += self.dw1
        dw2 += self.dw2
        dw3 += self.dw3
        dw4 += self.dw4
        dw5 += self.dw5
        dw6 += self.dw6
        db1 += self.db1
        db2 += self.db2
        db3 += self.db3
        
      m = X.shape[1]
      self.w1 -= learning_rate * dw1 / m
      self.w2 -= learning_rate * dw2 / m
      self.w3 -= learning_rate * dw3 / m
      self.w4 -= learning_rate * dw4 / m
      self.w5 -= learning_rate * dw5 / m
      self.w6 -= learning_rate * dw6 / m
      self.b1 -= learning_rate * db1 / m
      self.b2 -= learning_rate * db2 / m
      self.b3 -= learning_rate * db3 / m
      
      if display_loss:
        Y_pred = self.predict(X)
        loss[i] = mean_squared_error(Y_pred, Y)
    
    if display_loss:
      plt.plot(loss.values())
      plt.xlabel('Epochs')
      plt.ylabel('Mean Squared Error')
      plt.show()
      
  def predict(self, X):
    #predicting the results on unseen data
    Y_pred = []
    for x in X:
      y_pred = self.forward_pass(x)
      Y_pred.append(y_pred)
    return np.array(Y_pred)
</code></pre>
<p>The data was generated as follows :</p>
<pre><code>data, labels = make_blobs(n_samples=1000, centers=4, n_features=2, random_state=0)
labels_orig = labels
labels = np.mod(labels_orig, 2)
X_train, X_val, Y_train, Y_val = train_test_split(data, labels, stratify=labels, random_state=0)
</code></pre>
<p>When I ran the program yesterday, I had gotten a training accuracy of about 98% and a test accuracy of 94%. But when I ran it today, suddenly the accuracy dropped to 60-70%. I tried to scatter plot the result, and it looked like it behaved as if it were a single sigmoid instead of the Feed-Forward Network.</p>
<pre><code>ffn = FirstFFNetwork()
#train the model on the data
ffn.fit(X_train, Y_train, epochs=2000, learning_rate=.01, display_loss=False)
#predictions
Y_pred_train = ffn.predict(X_train)
Y_pred_binarised_train = (Y_pred_train &gt;= 0.5).astype(&quot;int&quot;).ravel()
Y_pred_val = ffn.predict(X_val)
Y_pred_binarised_val = (Y_pred_val &gt;= 0.5).astype(&quot;int&quot;).ravel()
accuracy_train_1 = accuracy_score(Y_pred_binarised_train, Y_train)
accuracy_val_1 = accuracy_score(Y_pred_binarised_val, Y_val)
#model performance
print(&quot;Training accuracy&quot;, round(accuracy_train_1, 2))
print(&quot;Validation accuracy&quot;, round(accuracy_val_1, 2)
</code></pre>
<p>I do not understand how this happened and cannot figure it out.</p>

CC BY-SA 4.0
2020-05-05T08:49:04.503Z
Could be your initial weights, if you keep more epochs and check the train/val loss pattern, you might be able to find out
SajanGohil
<neural-networks><deep-learning><feedforward-neural-networks><accuracy><sigmoid>
Accuracy dropped when I ran the program the second time
Eeshaan Jain
# Accuracy dropped when I ran the program the second time

(Asked by: username_0 on 2020-05-05)

I was following a tutorial about Feed-Forward Networks and wrote this code for a simple FFN :
class FirstFFNetwork:
  
  #intialize the parameters
  def __init__(self):
    self.w1 = np.random.randn()
    self.w2 = np.random.randn()
    self.w3 = np.random.randn()
    self.w4 = np.random.randn()
    self.w5 = np.random.randn()
    self.w6 = np.random.randn()
    self.b1 = 0
    self.b2 = 0
    self.b3 = 0
  
  def sigmoid(self, x):
    return 1.0/(1.0 + np.exp(-x))
  
  def forward_pass(self, x):
    #forward pass - preactivation and activation
    self.x1, self.x2 = x
    self.a1 = self.w1*self.x1 + self.w2*self.x2 + self.b1
    self.h1 = self.sigmoid(self.a1)
    self.a2 = self.w3*self.x1 + self.w4*self.x2 + self.b2
    self.h2 = self.sigmoid(self.a2)
    self.a3 = self.w5*self.h1 + self.w6*self.h2 + self.b3
    self.h3 = self.sigmoid(self.a3)
    return self.h3
  
  def grad(self, x, y):
    #back propagation
    self.forward_pass(x)
    
    self.dw5 = (self.h3-y) * self.h3*(1-self.h3) * self.h1
    self.dw6 = (self.h3-y) * self.h3*(1-self.h3) * self.h2
    self.db3 = (self.h3-y) * self.h3*(1-self.h3)
    
    self.dw1 = (self.h3-y) * self.h3*(1-self.h3) * self.w5 * self.h1*(1-self.h1) * self.x1
    self.dw2 = (self.h3-y) * self.h3*(1-self.h3) * self.w5 * self.h1*(1-self.h1) * self.x2
    self.db1 = (self.h3-y) * self.h3*(1-self.h3) * self.w5 * self.h1*(1-self.h1)
  
    self.dw3 = (self.h3-y) * self.h3*(1-self.h3) * self.w6 * self.h2*(1-self.h2) * self.x1
    self.dw4 = (self.h3-y) * self.h3*(1-self.h3) * self.w6 * self.h2*(1-self.h2) * self.x2
    self.db2 = (self.h3-y) * self.h3*(1-self.h3) * self.w6 * self.h2*(1-self.h2)
    
  
  def fit(self, X, Y, epochs=1, learning_rate=1, initialise=True, display_loss=False):
    
    # initialise w, b
    if initialise:
      self.w1 = np.random.randn()
      self.w2 = np.random.randn()
      self.w3 = np.random.randn()
      self.w4 = np.random.randn()
      self.w5 = np.random.randn()
      self.w6 = np.random.randn()
      self.b1 = 0
      self.b2 = 0
      self.b3 = 0
      
    if display_loss:
      loss = {}
    
    for i in tqdm_notebook(range(epochs), total=epochs, unit="epoch"):
      dw1, dw2, dw3, dw4, dw5, dw6, db1, db2, db3 = [0]*9
      for x, y in zip(X, Y):
        self.grad(x, y)
        dw1 += self.dw1
        dw2 += self.dw2
        dw3 += self.dw3
        dw4 += self.dw4
        dw5 += self.dw5
        dw6 += self.dw6
        db1 += self.db1
        db2 += self.db2
        db3 += self.db3
        
      m = X.shape[1]
      self.w1 -= learning_rate * dw1 / m
      self.w2 -= learning_rate * dw2 / m
      self.w3 -= learning_rate * dw3 / m
      self.w4 -= learning_rate * dw4 / m
      self.w5 -= learning_rate * dw5 / m
      self.w6 -= learning_rate * dw6 / m
      self.b1 -= learning_rate * db1 / m
      self.b2 -= learning_rate * db2 / m
      self.b3 -= learning_rate * db3 / m
      
      if display_loss:
        Y_pred = self.predict(X)
        loss[i] = mean_squared_error(Y_pred, Y)
    
    if display_loss:
      plt.plot(loss.values())
      plt.xlabel('Epochs')
      plt.ylabel('Mean Squared Error')
      plt.show()
      
  def predict(self, X):
    #predicting the results on unseen data
    Y_pred = []
    for x in X:
      y_pred = self.forward_pass(x)
      Y_pred.append(y_pred)
    return np.array(Y_pred)

The data was generated as follows :
data, labels = make_blobs(n_samples=1000, centers=4, n_features=2, random_state=0)
labels_orig = labels
labels = np.mod(labels_orig, 2)
X_train, X_val, Y_train, Y_val = train_test_split(data, labels, stratify=labels, random_state=0)

When I ran the program yesterday, I had gotten a training accuracy of about 98% and a test accuracy of 94%. But when I ran it today, suddenly the accuracy dropped to 60-70%. I tried to scatter plot the result, and it looked like it behaved as if it were a single sigmoid instead of the Feed-Forward Network.
ffn = FirstFFNetwork()
#train the model on the data
ffn.fit(X_train, Y_train, epochs=2000, learning_rate=.01, display_loss=False)
#predictions
Y_pred_train = ffn.predict(X_train)
Y_pred_binarised_train = (Y_pred_train >= 0.5).astype("int").ravel()
Y_pred_val = ffn.predict(X_val)
Y_pred_binarised_val = (Y_pred_val >= 0.5).astype("int").ravel()
accuracy_train_1 = accuracy_score(Y_pred_binarised_train, Y_train)
accuracy_val_1 = accuracy_score(Y_pred_binarised_val, Y_val)
#model performance
print("Training accuracy", round(accuracy_train_1, 2))
print("Validation accuracy", round(accuracy_val_1, 2)

I do not understand how this happened and cannot figure it out.

**Tags:** neural-networks, deep-learning, feedforward-neural-networks, accuracy, sigmoid

> **Comments:**
> **username_1**: Could be your initial weights, if you keep more epochs and check the train/val loss pattern, you might be able to find out  

---

## 1 answers

### Answer by username_2 on 2020-05-05. Score: 3

* It is common during the training of **Neural Networks** for **accuracy to improve for a while** and **then get worse** -- in general, This is caused by **over-fitting.** It's also fairly common for the Neural Network to **"get UNLUCKY** and get knocked into a **BAD sectors of parameter space** corresponding to a sudden decrease in accuracy -- sometimes it can recover from this quickly, but sometimes not.

[![image](https://i.sstatic.net/cFiA1.jpg)](https://i.sstatic.net/cFiA1.jpg)

* In general, lowering your **learning rate** is a **good approach** to this kind of problem. Also, setting a **learning rate schedule** like FactorScheduler can help you achieve **more stable convergence** by **lowering the learning rate every few epochs**. In fact, this can sometimes cover up mistakes in picking an initial learning rate that is too high.
* you can try using mini-batches.
* The error (Entropy) with log functions must be used **precisely**.

---

* [What are some possible reasons why training accuracy would decrease over time?](https://www.quora.com/What-are-some-possible-reasons-why-training-accuracy-would-decrease-over-time)
* Check out in this [paper](https://arxiv.org/abs/1506.01186) a novel way to change the learning rate cyclically.


---

