Answers.Body: <p>If you ruled out leakage completely read this observation about double decent <a href="https://openai.com/blog/deep-double-descent/" rel="nofollow noreferrer">https://openai.com/blog/deep-double-descent/</a>
  This Blogpost from openAI shares the observation that the validation loss can decrease again even after initially increasing (which is typically a sign for the start of overfitting, e.g. in early stopping).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-30T07:02:07.947Z
Answers.LastActivityDate: 2021-10-30T07:02:07.947Z
Answers.UserDisplayName: Ggjj11
Answers.Body: <p>You did a great job at this...</p>
  <p>You can use the Tensorflow’s LogSumExp built-in function to avoid numerical problems. This routine prevents over/under flow issues that may occur when LogSumExp encounters very extreme, either positive or negative values.</p>
  <p>You have sorted out this:</p>
  <p>There need to be <strong>Images from the generator.</strong> To these ones, the discriminator learns to classify as fake.</p>
  <p><strong>Real images with labels.</strong> These are image label pairs like in any regular supervised classification problem.</p>
  <p><strong>Real images without labels.</strong> For those, the classifier only learns that these images are real.</p>
  <p>I would recommend you visit this link:
  <strong><a href="https://towardsdatascience.com/semi-supervised-learning-with-gans-9f3cb128c5e" rel="nofollow noreferrer">Semi-supervised learning with Generative Adversarial Networks (GANs)</a></strong></p>
  <p>Good luck.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-30T06:18:24.280Z
Answers.LastActivityDate: 2021-10-30T06:24:48.103Z
Answers.LastEditDate: 2021-10-30T06:24:48.103Z
Answers.UserDisplayName: Okechukwu Ezekiel
Body: <p>I am training a semi-supervised GAN network using data from multiple subjects. I separated the labeled and unlabeled data based on my subjects, so there is no leakage, while having much more unlabeled data than labeled data. After few epochs training accuracy hits 100% which normally indicates overfitting, however the performance on the validation and test sets keeps increasing for 200-300 epochs. Is this considered overfitting and is there an explanation for this behavior?</p>
  <p><a href="https://i.sstatic.net/jDt7s.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jDt7s.png" alt="Train and validation curves during training." /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-14T03:30:49.240Z
Comments.Text: There is a lot of variables here which can help answer this. The main one: How are you measuring accuracy? For example, if you measure a correct prediction as above 0.5 when positive and below 0.5 when negative, the accuracy could be 100% well before overfitting, as the network still has a lot of room to approach 1 and 0.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-14T11:50:36.130Z
Comments.Text: I am in a multiclass setting with k=12 classes and I use argmax on the output logits, and compare them to real labels using sklearn's accuracy function. I am also computing the loss, for example in one epoch i had 0.4 for validation and 0.01 for training, and later training loss drops to 1e-5 while validation loss oscillates around 1, with no noticable impact on the accuracy score.
Comments.UserDisplayName: Hazar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-15T01:05:08.120Z
Comments.Text: How does it do on the test set? This does sound like it's overfitting based on the loss. I would triple check you are definitely running the same version of the network that's producing this loss when computing accuracy before continuing, because it does sound quite strange that loss would increase by over 100% with accuracy also increasing at the same time
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-16T02:06:11.917Z
Comments.Text: Same on the test set, performances kept increasing even after signs of overfitting on the training set. For example in a recent model, I obtained 100% acc, 0.0016 loss for training, around 93% acc and 0.4 loss for both validation and test sets that came from different subjects.
Comments.UserDisplayName: Hazar
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-16T11:34:29.537Z
Comments.Text: Can you maybe post a graph of the loss and accuracy for training and validation? That could help see what's going on here more exactly.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-24T14:17:36.463Z
Comments.Text: I just added a graph.
Comments.UserDisplayName: Hazar
Tags: <generative-adversarial-networks><overfitting><semi-supervised-learning>
Title: Validation set performance increasing even after seemingly overfit on training set
UserDisplayName: Hazar
ThreadText: # Validation set performance increasing even after seemingly overfit on training set
  
  (Asked by: username_0 on 2021-10-13)
  
  I am training a semi-supervised GAN network using data from multiple subjects. I separated the labeled and unlabeled data based on my subjects, so there is no leakage, while having much more unlabeled data than labeled data. After few epochs training accuracy hits 100% which normally indicates overfitting, however the performance on the validation and test sets keeps increasing for 200-300 epochs. Is this considered overfitting and is there an explanation for this behavior?
  
  [![Train and validation curves during training.](https://i.sstatic.net/jDt7s.png)](https://i.sstatic.net/jDt7s.png)
  
  **Tags:** generative-adversarial-networks, overfitting, semi-supervised-learning
  
  > **Comments:**
  > **username_1**: There is a lot of variables here which can help answer this. The main one: How are you measuring accuracy? For example, if you measure a correct prediction as above 0.5 when positive and below 0.5 when negative, the accuracy could be 100% well before overfitting, as the network still has a lot of room to approach 1 and 0.  
  > **username_0**: I am in a multiclass setting with k=12 classes and I use argmax on the output logits, and compare them to real labels using sklearn's accuracy function. I am also computing the loss, for example in one epoch i had 0.4 for validation and 0.01 for training, and later training loss drops to 1e-5 while validation loss oscillates around 1, with no noticable impact on the accuracy score.  
  > **username_1**: How does it do on the test set? This does sound like it's overfitting based on the loss. I would triple check you are definitely running the same version of the network that's producing this loss when computing accuracy before continuing, because it does sound quite strange that loss would increase by over 100% with accuracy also increasing at the same time  
  > **username_0**: Same on the test set, performances kept increasing even after signs of overfitting on the training set. For example in a recent model, I obtained 100% acc, 0.0016 loss for training, around 93% acc and 0.4 loss for both validation and test sets that came from different subjects.  
  > **username_1**: Can you maybe post a graph of the loss and accuracy for training and validation? That could help see what's going on here more exactly.  
  > **username_0**: I just added a graph.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-10-30. Score: 1
  
  If you ruled out leakage completely read this observation about double decent <https://openai.com/blog/deep-double-descent/>
  This Blogpost from openAI shares the observation that the validation loss can decrease again even after initially increasing (which is typically a sign for the start of overfitting, e.g. in early stopping).
  
  
  ---
  
  ### Answer by username_3 on 2021-10-30. Score: 1
  
  You did a great job at this...
  
  You can use the Tensorflow’s LogSumExp built-in function to avoid numerical problems. This routine prevents over/under flow issues that may occur when LogSumExp encounters very extreme, either positive or negative values.
  
  You have sorted out this:
  
  There need to be **Images from the generator.** To these ones, the discriminator learns to classify as fake.
  
  **Real images with labels.** These are image label pairs like in any regular supervised classification problem.
  
  **Real images without labels.** For those, the classifier only learns that these images are real.
  
  I would recommend you visit this link:
  **[Semi-supervised learning with Generative Adversarial Networks (GANs)](https://towardsdatascience.com/semi-supervised-learning-with-gans-9f3cb128c5e)**
  
  Good luck.
  
  
  ---
  
  