Answers.Body: <p>There's a few questions here so I'll address them in turn:</p>
  <ol>
  <li><p><strong>What to do with the results of a K-Fold Cross Validation?</strong> Generally, a Cross Validation (<a href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)" rel="nofollow noreferrer">or any other form of data partitioning</a>) is used as part of a <a href="https://en.wikipedia.org/wiki/Hyperparameter_optimization" rel="nofollow noreferrer">hyper-parameter seach</a>. Your final model would then be the trained using the best hyper-parameters found during the search, and all of the available training data. This final model should then be tested on a held-out set of testing data, which was not using in any stage of training,  to ensure no leakage of information.</p>
  </li>
  <li><p><strong>How to reduce the Loss?</strong> This is a tough question and there is no one direct answer that works in every situation. However improving the performance of a model can be crudely summaraised as: <em>better data</em>, <em>better model</em>, and <em>more regularisation</em>.</p>
  <p>Aquiring <em>better data</em> in many contexts is <a href="https://doi.org/10.1145/3394486.3406477" rel="nofollow noreferrer">often &quot;more data&quot;</a>, but there are analyses that suggest data cleaning and rebalancing can cause signficant improvements, but <a href="http://arxiv.org/pdf/1904.09483" rel="nofollow noreferrer">it's hard to measure these improvements</a>. These must be carried out with care to avoid leaking information between the training/testing sets. It is also almost always the hardest thing to do!</p>
  <p><em>Better models</em> can be achieved by changing your architecture or <a href="https://en.wikipedia.org/wiki/Ensemble_learning" rel="nofollow noreferrer">ensembling multiple models</a>. You've mentioned you're using 3D CNNs on video input, so possible <em>better model</em> options while keeping the same overall achitecture are changing the number of layers and experimenting with different shapes of convolutional kernel.</p>
  <p>Finally, <em>more regularisation</em> usually helps avoid overfitting and improves your model's ability to generalise to unseen data. This can be achieved by applying layers such as <a href="https://stats.stackexchange.com/questions/240305/where-should-i-place-dropout-layers-in-a-neural-network">Dropout</a>, <a href="https://en.wikipedia.org/wiki/Batch_normalization" rel="nofollow noreferrer">Batch Normalisation</a>, or applying a <a href="https://stats.stackexchange.com/questions/316961/l2-regularization-constant">regularisation constraint to your model's weights</a>.</p>
  </li>
  <li><p><strong>Is it overfitting?</strong> Probably, but that's not a death knell for your modelling approach <em>per se</em>. Your graphs show that for these K-fold splits, the performance of the model on the validation set stabilises out as worse than the performance on the training set. If we can assume that the distribution of targets in these validation sets is the same as in the training set, then the model has measurably learned patterns that are unique to the training sets. Thus, overfitting.</p>
  <p>However, it's worth checking that this assumption of equal distributions is true - and if not I'd recommend using <strong>Stratified K-Fold Cross Validation</strong> to ensure that it is true for future experiments to make a fair comparison.</p>
  </li>
  </ol>
  <p>If you re-run with difference architectures, regularisation and stratification (if needed) then the generalised performance of the model may improve. There's a lot of fun experimentation to try from here, and I wish you a healthy dose luck and patience! :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-04T13:31:25.697Z
Answers.LastActivityDate: 2022-10-04T13:31:25.697Z
Answers.UserDisplayName: James Ashford
Body: <p>hi i'm new in this field. I am trying to do a video classification project by using 3DCNN and I plotted the loss curves &amp; accuracy curves. I have some questions.</p>
  <ol>
  <li>i'm using kfold Cross validation. Should i save the parameters after every fold and choose the best one?</li>
  <li>how to reduce the loss?</li>
  <li>is it overfitting?</li>
  </ol>
  <p>one of the fold</p>
  <p><a href="https://i.sstatic.net/YPmak.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YPmak.png" alt="accuracy" /></a></p>
  <p><a href="https://i.sstatic.net/KPFbf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KPFbf.png" alt="loss" /></a></p>
  <p>another fold</p>
  <p><a href="https://i.sstatic.net/mgE41.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mgE41.png" alt="accuracy" /></a></p>
  <p><a href="https://i.sstatic.net/rIys0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rIys0.png" alt="loss" /></a></p>
  
Tags: <neural-networks><machine-learning><algorithm><3d-convolution>
Title: Is it overfitting?
UserDisplayName: david
ThreadText: # Is it overfitting?
  
  (Asked by: username_0 on 2022-09-20)
  
  hi i'm new in this field. I am trying to do a video classification project by using 3DCNN and I plotted the loss curves & accuracy curves. I have some questions.
  
  1. i'm using kfold Cross validation. Should i save the parameters after every fold and choose the best one?
  2. how to reduce the loss?
  3. is it overfitting?
  
  one of the fold
  
  [![accuracy](https://i.sstatic.net/YPmak.png)](https://i.sstatic.net/YPmak.png)
  
  [![loss](https://i.sstatic.net/KPFbf.png)](https://i.sstatic.net/KPFbf.png)
  
  another fold
  
  [![accuracy](https://i.sstatic.net/mgE41.png)](https://i.sstatic.net/mgE41.png)
  
  [![loss](https://i.sstatic.net/rIys0.png)](https://i.sstatic.net/rIys0.png)
  
  **Tags:** neural-networks, machine-learning, algorithm, 3d-convolution
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-04. Score: 1
  
  There's a few questions here so I'll address them in turn:
  
  1. **What to do with the results of a K-Fold Cross Validation?** Generally, a Cross Validation ([or any other form of data partitioning](https://en.wikipedia.org/wiki/Cross-validation_(statistics))) is used as part of a [hyper-parameter seach](https://en.wikipedia.org/wiki/Hyperparameter_optimization). Your final model would then be the trained using the best hyper-parameters found during the search, and all of the available training data. This final model should then be tested on a held-out set of testing data, which was not using in any stage of training, to ensure no leakage of information.
  2. **How to reduce the Loss?** This is a tough question and there is no one direct answer that works in every situation. However improving the performance of a model can be crudely summaraised as: *better data*, *better model*, and *more regularisation*.
  
     Aquiring *better data* in many contexts is [often "more data"](https://doi.org/10.1145/3394486.3406477), but there are analyses that suggest data cleaning and rebalancing can cause signficant improvements, but [it's hard to measure these improvements](http://arxiv.org/pdf/1904.09483). These must be carried out with care to avoid leaking information between the training/testing sets. It is also almost always the hardest thing to do!
  
     *Better models* can be achieved by changing your architecture or [ensembling multiple models](https://en.wikipedia.org/wiki/Ensemble_learning). You've mentioned you're using 3D CNNs on video input, so possible *better model* options while keeping the same overall achitecture are changing the number of layers and experimenting with different shapes of convolutional kernel.
  
     Finally, *more regularisation* usually helps avoid overfitting and improves your model's ability to generalise to unseen data. This can be achieved by applying layers such as [Dropout](https://stats.stackexchange.com/questions/240305/where-should-i-place-dropout-layers-in-a-neural-network), [Batch Normalisation](https://en.wikipedia.org/wiki/Batch_normalization), or applying a [regularisation constraint to your model's weights](https://stats.stackexchange.com/questions/316961/l2-regularization-constant).
  3. **Is it overfitting?** Probably, but that's not a death knell for your modelling approach *per se*. Your graphs show that for these K-fold splits, the performance of the model on the validation set stabilises out as worse than the performance on the training set. If we can assume that the distribution of targets in these validation sets is the same as in the training set, then the model has measurably learned patterns that are unique to the training sets. Thus, overfitting.
  
     However, it's worth checking that this assumption of equal distributions is true - and if not I'd recommend using **Stratified K-Fold Cross Validation** to ensure that it is true for future experiments to make a fair comparison.
  
  If you re-run with difference architectures, regularisation and stratification (if needed) then the generalised performance of the model may improve. There's a lot of fun experimentation to try from here, and I wish you a healthy dose luck and patience! :)
  
  
  ---
  
  