Answers.Body: <p>It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001
  ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train your model better. The learning rate is about steps to change weights, in this plot you see that the validation loss is not changing with an optimization goal</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-08T10:51:47.243Z
Answers.Text: Hi your method worked, thank you! I'm trying to get a better insight into why it is. I'm plotting the trainable parameters on TensorBoard, do you have any recommendations as to what I should look out for? Also I have a follow up post [here](https://ai.stackexchange.com/questions/18474/cant-reproduce-model-fit-with-gradienttape). @SahaTib
Answers.UserDisplayName: Jack-P
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-09T22:29:32.360Z
Answers.Text: @Jack-P glad to hear that, check this out:https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/ it may help
Answers.UserDisplayName: SahaTib
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T07:57:03.217Z
Answers.Text: Thanks for the resource! I’m actually trying to understand why SGD was able to overfit the model (I’m following the general advice to first overfit a model to make sure it works) while Adam couldn’t as evident from the high training loss. Is it perhaps because it’s stuck at a saddle point or a local minima but the stochastic nature of SGD was able to escape? @SahaTib
Answers.UserDisplayName: Jack-P
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-25T11:03:23.483Z
Answers.LastActivityDate: 2020-02-25T16:07:03.897Z
Answers.LastEditDate: 2020-02-25T16:07:03.897Z
Answers.UserDisplayName: SahaTib
Answers.Body: <p>Are you using BinaryCrossEntropy through tensorflow? If so, check if you are using the logits argument. I am using from_logits=True .It is not similar to the original BinaryCrossEntropy loss.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-26T22:28:55.080Z
Answers.LastActivityDate: 2021-04-26T16:18:41.170Z
Answers.LastEditDate: 2021-04-26T16:18:41.170Z
Answers.UserDisplayName: Tae Soo Kim
Body: <p>I’m trying to debug my neural network (BERT fine-tuning) trained for natural language inference with binary classification of either entailment or contradiction. I've trained it for 80 epochs and its converging on ~0.68. Why isn't it getting any lower?</p>
  
  <p>Thanks in advance!</p>
  
  <hr>
  
  <p>Neural Network Architecture:</p>
  
  <p><a href="https://i.sstatic.net/3VAyw.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3VAyw.jpg" alt="Architecture"></a></p>
  
  <p>Training details:</p>
  
  <ul>
  <li>Loss function: Binary cross entropy</li>
  <li>Batch size: 8</li>
  <li>Optimizer: Adam (learning rate = 0.001)</li>
  <li>Framework: Tensorflow 2.0.1</li>
  <li>Pooled embeddings used from BERT output.</li>
  <li>BERT parameters are not frozen.</li>
  </ul>
  
  <p>Dataset:</p>
  
  <ul>
  <li>10,000 samples</li>
  <li>balanced dataset (5k each for entailment and contradiction)</li>
  <li>dataset is a subset of data mined from wikipedia.</li>
  <li>Claim example: <em>"'History of art includes architecture, dance, sculpture, music, painting, poetry literature, theatre, narrative, film, photography and graphic arts.'"</em></li>
  <li>Evidence example: <em>"The subsequent expansion of the list of principal arts in the 20th century reached to nine : architecture , dance , sculpture , music , painting , poetry -LRB- described broadly as a form of literature with aesthetic purpose or function , which also includes the distinct genres of theatre and narrative -RRB- , film , photography and graphic arts ."</em></li>
  </ul>
  
  <p>Dataset preprocessing: </p>
  
  <ul>
  <li>Used [SEP] to separate the two sentences instead of using separate embeddings via 2 BERT layers. (Hence, segment ids are computed as such)</li>
  <li>BERT's <a href="https://github.com/google-research/bert/blob/master/tokenization.py" rel="nofollow noreferrer">FullTokenizer</a> for tokenization.</li>
  <li>Truncated to a maximum sequence length of 64.</li>
  </ul>
  
  <p>See below for a graph of the training history. (Red = train_loss, Blue = val_loss)</p>
  
  <p><a href="https://i.sstatic.net/h8Hlr.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/h8Hlr.png" alt="0.68 convergence"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-05T16:23:46.097Z
Comments.Text: I ran into a similar problem and my issue was that I had a Sigmoid layer at the end of the model (that I thought I had removed). So I had `Sigmoid->BCEWithLogitsLoss`, effectively doing two sigmoids after another. Funnily enough, it was working anyway when I was doing `Sigmoid->Sigmoid->L1`... so not sure how much of an answer this is. Worth double-checking if anyone has the same issue though.
Comments.UserDisplayName: lucidbrot
Tags: <deep-learning><natural-language-processing><tensorflow><training><bert>
Title: Why is my loss (binary cross entropy) converging on ~0.6? (Task: Natural Language Inference)
UserDisplayName: Jack-P
ThreadText: # Why is my loss (binary cross entropy) converging on ~0.6? (Task: Natural Language Inference)
  
  (Asked by: username_0 on 2020-02-25)
  
  I’m trying to debug my neural network (BERT fine-tuning) trained for natural language inference with binary classification of either entailment or contradiction. I've trained it for 80 epochs and its converging on ~0.68. Why isn't it getting any lower?
  
  Thanks in advance!
  
  ---
  
  Neural Network Architecture:
  
  [![Architecture](https://i.sstatic.net/3VAyw.jpg)](https://i.sstatic.net/3VAyw.jpg)
  
  Training details:
  
  * Loss function: Binary cross entropy
  * Batch size: 8
  * Optimizer: Adam (learning rate = 0.001)
  * Framework: Tensorflow 2.0.1
  * Pooled embeddings used from BERT output.
  * BERT parameters are not frozen.
  
  Dataset:
  
  * 10,000 samples
  * balanced dataset (5k each for entailment and contradiction)
  * dataset is a subset of data mined from wikipedia.
  * Claim example: *"'History of art includes architecture, dance, sculpture, music, painting, poetry literature, theatre, narrative, film, photography and graphic arts.'"*
  * Evidence example: *"The subsequent expansion of the list of principal arts in the 20th century reached to nine : architecture , dance , sculpture , music , painting , poetry -LRB- described broadly as a form of literature with aesthetic purpose or function , which also includes the distinct genres of theatre and narrative -RRB- , film , photography and graphic arts ."*
  
  Dataset preprocessing:
  
  * Used [SEP] to separate the two sentences instead of using separate embeddings via 2 BERT layers. (Hence, segment ids are computed as such)
  * BERT's [FullTokenizer](https://github.com/google-research/bert/blob/master/tokenization.py) for tokenization.
  * Truncated to a maximum sequence length of 64.
  
  See below for a graph of the training history. (Red = train\_loss, Blue = val\_loss)
  
  [![0.68 convergence](https://i.sstatic.net/h8Hlr.png)](https://i.sstatic.net/h8Hlr.png)
  
  **Tags:** deep-learning, natural-language-processing, tensorflow, training, bert
  > **Comments:**
  > **username_1**: I ran into a similar problem and my issue was that I had a Sigmoid layer at the end of the model (that I thought I had removed). So I had `Sigmoid->BCEWithLogitsLoss`, effectively doing two sigmoids after another. Funnily enough, it was working anyway when I was doing `Sigmoid->Sigmoid->L1`... so not sure how much of an answer this is. Worth double-checking if anyone has the same issue though.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-02-25. Score: 2 | Accepted answer
  
  It seems to be overfitting and your model is not learning. Try SGD optimizer with a learning rate of 0.001
  ADAM optimizer will give you a soon overfitting, and decreasing the learning rate will train your model better. The learning rate is about steps to change weights, in this plot you see that the validation loss is not changing with an optimization goal
  
  > **Comments:**
  > **username_0**: Hi your method worked, thank you! I'm trying to get a better insight into why it is. I'm plotting the trainable parameters on TensorBoard, do you have any recommendations as to what I should look out for? Also I have a follow up post [here](https://ai.stackexchange.com/questions/18474/cant-reproduce-model-fit-with-gradienttape). @SahaTib  
  > **username_2**: @Jack-P glad to hear that, check this out:https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/ it may help  
  > **username_0**: Thanks for the resource! I’m actually trying to understand why SGD was able to overfit the model (I’m following the general advice to first overfit a model to make sure it works) while Adam couldn’t as evident from the high training loss. Is it perhaps because it’s stuck at a saddle point or a local minima but the stochastic nature of SGD was able to escape? @SahaTib  
  
  ---
  
  ### Answer by username_3 on 2021-01-26. Score: 0
  
  Are you using BinaryCrossEntropy through tensorflow? If so, check if you are using the logits argument. I am using from\_logits=True .It is not similar to the original BinaryCrossEntropy loss.
  
  
  ---
  
  