Body: <p>I trained a recurrent neural network (if it matters - it contains three CuDNNLSTM cells and 3 Dense layers, Dropout = 0.2). The result of data preparation is one array of ~330.000 sequences. Each contains 256 time steps and 24 features in each time step. This array is normalized, shuffled and balanced. Then it is split into two arrays - train array contains 90% of data (so ~297k) and validation array contains ~10%.</p>
  
  <p>During training process (Adam optimizer, 128 or 256 batches) max accuracy of validation data set is 90%. Epoch accuracy is 94%.</p>
  
  <p>Then I run my additional validation test script with more realistic, unbalanced data set. The accuracy of test is just ~50%. I checked this second test if it is correct and there is no errors in the code, but when I run it on data, that was included in the training set, the accuracy was 87%, so it looks good.</p>
  
  <p>What is going on? I suppose, that wrong architecture is used.</p>
  
  <p>Here is a graph of epoch accuracy during training:
  <a href="https://i.sstatic.net/aBGPz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/aBGPz.png" alt="Epoch accuracy"></a></p>
  
  <p>Here is a graph of validation accuracy during training
  <a href="https://i.sstatic.net/CZAw1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/CZAw1.png" alt="validation accuracy"></a></p>
  
  <p>Thank you for support.</p>
  
  <p><strong>UPDATE:</strong></p>
  
  <p>Today I trained the network one more time. I used the unbalanced data from second test as validation data set in traininge process. You can see the results below.I stopped training after 29 epochs. </p>
  
  <p><strong>Blue:</strong> balanced valitadion data set</p>
  
  <p><strong>Orange:</strong> unbalanced validation data set</p>
  
  <p><strong>Epoch accuracy:</strong>
  <a href="https://i.sstatic.net/lXjFm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/lXjFm.png" alt="Epoch accuracy"></a></p>
  
  <p><strong>Validation accuracy</strong>
  <a href="https://i.sstatic.net/eKOBj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/eKOBj.png" alt="Validation accuracy"></a></p>
  
  <p><strong>Validation loss</strong>
  <a href="https://i.sstatic.net/FB5r0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FB5r0.png" alt="Validation loss"></a></p>
  
  <p>It doesn't look good at all.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-07T09:36:48.187Z
Comments.Text: Looks like common generalization problem, network overfit on the 1st dataset. Try to train on realistic datset or augment 1st with random transformations if you can't get more training data.
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-07T09:44:38.117Z
Comments.Text: Thank you for your answer. Today i run learning one more time with the same training data set but validation data from the second test. In a few hours we can verify your remark.
Comments.UserDisplayName: ketzul
Tags: <neural-networks><tensorflow><long-short-term-memory><intelligence-testing>
Title: RNN: Different test results on balanced and unbalanced data
UserDisplayName: ketzul
ThreadText: # RNN: Different test results on balanced and unbalanced data
  
  (Asked by: username_0 on 2019-04-06)
  
  I trained a recurrent neural network (if it matters - it contains three CuDNNLSTM cells and 3 Dense layers, Dropout = 0.2). The result of data preparation is one array of ~330.000 sequences. Each contains 256 time steps and 24 features in each time step. This array is normalized, shuffled and balanced. Then it is split into two arrays - train array contains 90% of data (so ~297k) and validation array contains ~10%.
  
  During training process (Adam optimizer, 128 or 256 batches) max accuracy of validation data set is 90%. Epoch accuracy is 94%.
  
  Then I run my additional validation test script with more realistic, unbalanced data set. The accuracy of test is just ~50%. I checked this second test if it is correct and there is no errors in the code, but when I run it on data, that was included in the training set, the accuracy was 87%, so it looks good.
  
  What is going on? I suppose, that wrong architecture is used.
  
  Here is a graph of epoch accuracy during training:
  [![Epoch accuracy](https://i.sstatic.net/aBGPz.png)](https://i.sstatic.net/aBGPz.png)
  
  Here is a graph of validation accuracy during training
  [![validation accuracy](https://i.sstatic.net/CZAw1.png)](https://i.sstatic.net/CZAw1.png)
  
  Thank you for support.
  
  **UPDATE:**
  
  Today I trained the network one more time. I used the unbalanced data from second test as validation data set in traininge process. You can see the results below.I stopped training after 29 epochs.
  
  **Blue:** balanced valitadion data set
  
  **Orange:** unbalanced validation data set
  
  **Epoch accuracy:**
  [![Epoch accuracy](https://i.sstatic.net/lXjFm.png)](https://i.sstatic.net/lXjFm.png)
  
  **Validation accuracy**
  [![Validation accuracy](https://i.sstatic.net/eKOBj.png)](https://i.sstatic.net/eKOBj.png)
  
  **Validation loss**
  [![Validation loss](https://i.sstatic.net/FB5r0.png)](https://i.sstatic.net/FB5r0.png)
  
  It doesn't look good at all.
  
  **Tags:** neural-networks, tensorflow, long-short-term-memory, intelligence-testing
  
  > **Comments:**
  > **username_1**: Looks like common generalization problem, network overfit on the 1st dataset. Try to train on realistic datset or augment 1st with random transformations if you can't get more training data.  
  > **username_0**: Thank you for your answer. Today i run learning one more time with the same training data set but validation data from the second test. In a few hours we can verify your remark.  
  
  ---
  
  ## 0 answers
  
  