Answers.Body: <p>This looks like overfitting. You can try <a href="https://en.wikipedia.org/wiki/Early_stopping" rel="nofollow noreferrer">stop training earlier</a> by using a validation dataset to prevent this, or you can try other regularization effects such as weight-decay, dropout etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-22T03:35:34.167Z
Answers.LastActivityDate: 2021-11-22T03:35:34.167Z
Answers.UserDisplayName: SpiderRico
Body: <p><a href="https://i.sstatic.net/AidUJ.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AidUJ.png" alt="enter image description here" /></a></p>
  <p>I want to train a neural network model with the arcface loss function and try to combine it with domain adaption. But when the training process continues, I find the test accuracy first increases and then decreases, the model cannot reach convergence. I chose the office31 dataset, and the feature_extractor was resnet50.</p>
  <p>I want to know if it is caused by my code, or by my loss function</p>
  <p>The arcface function was set as</p>
  <pre><code>def Arc_pred(cosine, s=64.0, m=0.1):
      cosine = cosine / s
      thea = torch.acos(cosine)
      top = torch.exp(torch.cos(thea + m) * s)
      _top = torch.exp(torch.cos(thea) * s)
      bottom = torch.sum(torch.exp(cosine * s), dim=1).view(-1, 1)
      divide = (top / (bottom - _top + top)) + 1e-10
      return divide
  </code></pre>
  <p>and my total loss function was set as</p>
  <pre><code>total_loss = 0.1*target_entropy_loss + label_loss + arc_loss + discriminator_loss
  </code></pre>
  <p>In that, the <code>target_entropy_loss</code> tries to make the decision boundary cross the sparsest sample area，<code>label_loss</code> was the classification loss, <code>discriminator_loss</code> was a domain adaptation loss function.</p>
  <p>I tried to set a learning rate schedule for my experiment, it seems it did not work. So, could it be caused by my loss function?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-25T01:05:44.073Z
Comments.Text: Hello. From the plot you're showing us, it's not clear what is the highest test accuracy reached after it drops. As suggested in the answer below, you could try early stopping, but if the top test accuracy is too low, you may try something else first. Could you please also tell us how long have you been training the model for, at which epoch the test accuracy drops and how large are your training and test datasets.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-27T01:39:55.480Z
Comments.Text: I tried to train the network by 10000 iterations,the highest test accuracy reached at about 91%.Besides overfitting,I want to know whether the SGD optimizer I chosed for the GAN part also could cause the decrease.
Comments.UserDisplayName: klayoe
Tags: <machine-learning><objective-functions><unsupervised-learning><performance><accuracy>
Title: Test accuracy decreases during my train process
UserDisplayName: klayoe
ThreadText: # Test accuracy decreases during my train process
  
  (Asked by: username_0 on 2021-11-22)
  
  [![enter image description here](https://i.sstatic.net/AidUJ.png)](https://i.sstatic.net/AidUJ.png)
  
  I want to train a neural network model with the arcface loss function and try to combine it with domain adaption. But when the training process continues, I find the test accuracy first increases and then decreases, the model cannot reach convergence. I chose the office31 dataset, and the feature\_extractor was resnet50.
  
  I want to know if it is caused by my code, or by my loss function
  
  The arcface function was set as
  
  ```
  def Arc_pred(cosine, s=64.0, m=0.1):
      cosine = cosine / s
      thea = torch.acos(cosine)
      top = torch.exp(torch.cos(thea + m) * s)
      _top = torch.exp(torch.cos(thea) * s)
      bottom = torch.sum(torch.exp(cosine * s), dim=1).view(-1, 1)
      divide = (top / (bottom - _top + top)) + 1e-10
      return divide
  
  ```
  
  and my total loss function was set as
  
  ```
  total_loss = 0.1*target_entropy_loss + label_loss + arc_loss + discriminator_loss
  
  ```
  
  In that, the `target_entropy_loss` tries to make the decision boundary cross the sparsest sample area，`label_loss` was the classification loss, `discriminator_loss` was a domain adaptation loss function.
  
  I tried to set a learning rate schedule for my experiment, it seems it did not work. So, could it be caused by my loss function?
  
  **Tags:** machine-learning, objective-functions, unsupervised-learning, performance, accuracy
  
  > **Comments:**
  > **username_1**: Hello. From the plot you're showing us, it's not clear what is the highest test accuracy reached after it drops. As suggested in the answer below, you could try early stopping, but if the top test accuracy is too low, you may try something else first. Could you please also tell us how long have you been training the model for, at which epoch the test accuracy drops and how large are your training and test datasets.  
  > **username_0**: I tried to train the network by 10000 iterations,the highest test accuracy reached at about 91%.Besides overfitting,I want to know whether the SGD optimizer I chosed for the GAN part also could cause the decrease.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-11-22. Score: 1
  
  This looks like overfitting. You can try [stop training earlier](https://en.wikipedia.org/wiki/Early_stopping) by using a validation dataset to prevent this, or you can try other regularization effects such as weight-decay, dropout etc.
  
  
  ---
  
  