Answers.Body: <p>I'll try to answer on more general questions</p>
  <ol>
  <li>Is it ok that model performs better on validation, then on train?</li>
  </ol>
  <p>It's certainly fine if you use techniques like dropout or data augmentation and the difference is not that big. Because in case of dropout for train you use part of the network, and for validation the whole.</p>
  <ol start="2">
  <li>I'm suspicious my model is too good. What could I do?</li>
  </ol>
  <p>That a good point, because in my experience too good results sometime are reasoned by the flow in the training. The most common is data leakage, that's mean you give the model a way to fool around in unexpected and unwanted way.</p>
  <p>Let me give an example. Lets imagine you try to detect malicious traffic and you have a data about sessions and you split records on train/val. Let's imagine one of the features is IP address and you could have multiple sessions from same IP. But if same IP would be in both train and val for the crooked users, then model could just remember the 'bad' IP and thus getting high score on both train and val.</p>
  <p>So, to avoid it you need to think about things like that. If records could be group in some too meaningful way, make sure you put the whole group either in train or val. Another thing, that's it's usually good thing to split it by time as well, i.e. use for train records for 2018 and 2019 and for val for 2020. Thus you not only avoid data leakage, but you make sure your model is robust for future predictions.</p>
  <p>More general, try to understand, why you getting such good result with such simple model. Try to squeeze the data even smaller, drop some features to understand, when it would go done.</p>
  <p>Finally I would recommend the post by Andrej Karpathy, when he gave lots of really good advices on practical side of training NN. <a href="http://karpathy.github.io/2019/04/25/recipe/" rel="nofollow noreferrer">http://karpathy.github.io/2019/04/25/recipe/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-25T19:41:55.330Z
Answers.Text: great answer thanks a lot. Ill just note that indeed I did drop "ip addresses" exactly due to what you explained. I, however, did not drop port numbers - ill try to remove them and see.
Answers.UserDisplayName: nachofest
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-25T17:05:48.137Z
Answers.LastActivityDate: 2021-03-25T17:05:48.137Z
Answers.UserDisplayName: Kirill Fedyanin
Body: <p>I am aware of similar questions that have been asked, and I have gone through many. I want to bring my case to SE to understand better what my results are.</p>
  <p>I am working with a large dataset (around 75million records), but, for the purpose of testing techniques, I am actually using 2M records. I am working towards malicious traffic identification using NetFlow data. After employing some undersampling to have a balanced dataset according to my target variable (benign or attack) I have 1,240,950 of records in the training set and 310,238 in the validation set. Therefore I believe there is a good amount of data to train a Deep neural network properly.</p>
  <p>After using Yeo-Yohnsons transform and standardizing the data, I train the network with a very basic model:</p>
  <pre><code>def basem():    
      
      model = Sequential()
      
      model.add(Dense(25, input_dim=38))
      model.add(Activation(&quot;relu&quot;))
      
      model.add(Dense(50))
      model.add(Activation(&quot;relu&quot;))
      
      model.add(Dense(50))
      model.add(Activation(&quot;relu&quot;))
      
      model.add(Dense(25))
      model.add(Activation(&quot;relu&quot;))
      
      model.add(Dense(1, activation='sigmoid'))
      model.compile(loss='binary_crossentropy', optimizer=&quot;adam&quot;, metrics=['accuracy'])
      return model
  
  
  model_base = basem()
  model_base._name = 'base'
  
  history_base = model_base.fit(X_train, y_train, batch_size=2048, 
                      epochs=15, validation_data=(X_val,y_val), shuffle=True)
  </code></pre>
  <p>This gives me the following plot <a href="https://i.sstatic.net/C6mag.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/C6mag.png" alt="enter image description here" /></a></p>
  <p>It maybe because I am a newbie, but this plot looks too perfect. It is weird to see validation and training accuracy growing together, although I believe this is what we want right? But now I have the feeling it is overfitting. Therefore I use the model and a 5-fold cross validation to understand how well it generalizes. Results, mean accuracy and mean std(%), are:</p>
  <pre><code>test acc: 0.9816503485233088
  test_prec: 0.9840033637114158
  test_f1: 0.9816046990113001
  test_recall: 0.9792384866432975
  test_roc_auc: 0.9980004347946355
  
  Dev acc: 0.052931962886091546
  Dev prec: 0.2854656099314699
  Dev f1: 0.057228805478181974
  Dev recall: 0.3597811552056071
  Dev roc auc: 0.0036456892671197097
  </code></pre>
  <p>If I understand correctly, accuracy is high which is generally good and the standard deviation is very low for each metric, the highest being 0.359% for recall. Does this mean my model generalizes well?</p>
  <h3>Edit</h3>
  <p>Adding dropout (0.3) to each layer yields the following:</p>
  <p><a href="https://i.sstatic.net/TaFpX.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/TaFpX.png" alt="enter image description here" /></a></p>
  <p>Now, my validation accuracy is higher than my training. I can't make sense of any of this.</p>
  
Tags: <deep-learning><overfitting><accuracy><generalization><dropout>
Title: Is it possible that the model is overfitting when the training and validation accuracy increase?
UserDisplayName: nachofest
ThreadText: # Is it possible that the model is overfitting when the training and validation accuracy increase?
  
  (Asked by: username_0 on 2021-03-25)
  
  I am aware of similar questions that have been asked, and I have gone through many. I want to bring my case to SE to understand better what my results are.
  I am working with a large dataset (around 75million records), but, for the purpose of testing techniques, I am actually using 2M records. I am working towards malicious traffic identification using NetFlow data. After employing some undersampling to have a balanced dataset according to my target variable (benign or attack) I have 1,240,950 of records in the training set and 310,238 in the validation set. Therefore I believe there is a good amount of data to train a Deep neural network properly.
  After using Yeo-Yohnsons transform and standardizing the data, I train the network with a very basic model:
  def basem():    
      
      model = Sequential()
      
      model.add(Dense(25, input_dim=38))
      model.add(Activation("relu"))
      
      model.add(Dense(50))
      model.add(Activation("relu"))
      
      model.add(Dense(50))
      model.add(Activation("relu"))
      
      model.add(Dense(25))
      model.add(Activation("relu"))
      
      model.add(Dense(1, activation='sigmoid'))
      model.compile(loss='binary_crossentropy', optimizer="adam", metrics=['accuracy'])
      return model
  
  
  model_base = basem()
  model_base._name = 'base'
  
  history_base = model_base.fit(X_train, y_train, batch_size=2048, 
                      epochs=15, validation_data=(X_val,y_val), shuffle=True)
  
  This gives me the following plot 
  It maybe because I am a newbie, but this plot looks too perfect. It is weird to see validation and training accuracy growing together, although I believe this is what we want right? But now I have the feeling it is overfitting. Therefore I use the model and a 5-fold cross validation to understand how well it generalizes. Results, mean accuracy and mean std(%), are:
  test acc: 0.9816503485233088
  test_prec: 0.9840033637114158
  test_f1: 0.9816046990113001
  test_recall: 0.9792384866432975
  test_roc_auc: 0.9980004347946355
  
  Dev acc: 0.052931962886091546
  Dev prec: 0.2854656099314699
  Dev f1: 0.057228805478181974
  Dev recall: 0.3597811552056071
  Dev roc auc: 0.0036456892671197097
  
  If I understand correctly, accuracy is high which is generally good and the standard deviation is very low for each metric, the highest being 0.359% for recall. Does this mean my model generalizes well?
  Edit
  Adding dropout (0.3) to each layer yields the following:
  
  Now, my validation accuracy is higher than my training. I can't make sense of any of this.
  
  **Tags:** deep-learning, overfitting, accuracy, generalization, dropout
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-25. Score: 1
  
  I'll try to answer on more general questions
  
  1. Is it ok that model performs better on validation, then on train?
  
  It's certainly fine if you use techniques like dropout or data augmentation and the difference is not that big. Because in case of dropout for train you use part of the network, and for validation the whole.
  
  2. I'm suspicious my model is too good. What could I do?
  
  That a good point, because in my experience too good results sometime are reasoned by the flow in the training. The most common is data leakage, that's mean you give the model a way to fool around in unexpected and unwanted way.
  
  Let me give an example. Lets imagine you try to detect malicious traffic and you have a data about sessions and you split records on train/val. Let's imagine one of the features is IP address and you could have multiple sessions from same IP. But if same IP would be in both train and val for the crooked users, then model could just remember the 'bad' IP and thus getting high score on both train and val.
  
  So, to avoid it you need to think about things like that. If records could be group in some too meaningful way, make sure you put the whole group either in train or val. Another thing, that's it's usually good thing to split it by time as well, i.e. use for train records for 2018 and 2019 and for val for 2020. Thus you not only avoid data leakage, but you make sure your model is robust for future predictions.
  
  More general, try to understand, why you getting such good result with such simple model. Try to squeeze the data even smaller, drop some features to understand, when it would go done.
  
  Finally I would recommend the post by Andrej Karpathy, when he gave lots of really good advices on practical side of training NN. <http://karpathy.github.io/2019/04/25/recipe/>
  
  > **Comments:**
  > **username_0**: great answer thanks a lot. Ill just note that indeed I did drop "ip addresses" exactly due to what you explained. I, however, did not drop port numbers - ill try to remove them and see.  
  
  ---
  
  