Answers.Body: <p>Follow <a href="https://stackoverflow.com/questions/37044600/sudden-drop-in-accuracy-while-training-a-deep-neural-net">this answer</a> from StackOverflow, I think your problem related to the second cases where your loss gets Nan value.</p>
  <p>Maybe you should try to use the larger datatype (for example float 16 -&gt; float 32)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-14T10:45:37.910Z
Answers.Text: As I'm using tensorflow's built-in categorical cross-entropy, I think tf's team knows they should take good care of NaN, values. By the way, can regularization help in this situation or should I put weight constraints?
Answers.UserDisplayName: Sepehr Golestanian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-14T11:28:10.550Z
Answers.Text: We have to dig deeper into TF code to get the final conclusion. For the second question, I think yes, regularization is to solve the overfitting problem, here, before epoch 394, your validation accuracy is stable so I can say your model get overfit problem
Answers.UserDisplayName: CuCaRot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-14T12:51:35.697Z
Answers.Text: Thanks very much, I'll try this and update the question!
Answers.UserDisplayName: Sepehr Golestanian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-15T17:51:08.373Z
Answers.Text: @SepehrGolestanian So, why did you accept this answer? What was the problem in the end?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-16T01:35:41.340Z
Answers.Text: @nbro, you look like the admin of this community, I saw you carefully edit nearly all the post and ask in very details :smile:
Answers.UserDisplayName: CuCaRot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-16T09:16:20.287Z
Answers.Text: @Toby I am a [moderator](https://meta.stackexchange.com/q/75189/287113). I try to ask for clarification and details to make this site a better and more reliable place.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-16T09:31:46.857Z
Answers.Text: @nbro I see, glad this community has someone like you
Answers.UserDisplayName: CuCaRot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-16T10:40:09.867Z
Answers.Text: @Toby exactly! And I'm glad I found this community that has people like him in it!
Answers.UserDisplayName: Sepehr Golestanian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-17T04:37:59.607Z
Answers.Text: @SepehrGolestanian but back to his question, can my answer solve your problem?
Answers.UserDisplayName: CuCaRot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-17T12:59:33.243Z
Answers.Text: @Toby You mean the NaN thing or overfitting problem?
Answers.UserDisplayName: Sepehr Golestanian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-17T13:29:13.477Z
Answers.Text: I don't know, which one solves your problem :))
Answers.UserDisplayName: CuCaRot
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-14T10:17:35.587Z
Answers.LastActivityDate: 2020-12-14T10:17:35.587Z
Answers.UserDisplayName: CuCaRot
Answers.Body: <p>You have to keep the training loss value constant. Beyond that, your teaching technique or network model is not suitable for the job. I recommend that you are using MLP.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-11T21:12:22.277Z
Answers.LastActivityDate: 2023-03-11T21:20:38.290Z
Answers.LastEditDate: 2023-03-11T21:20:38.290Z
Answers.UserDisplayName: fkybrd
Body: <p>I was training a CNN model on TensorFlow. After a while I came back and saw this loss curve:</p>
  <p><a href="https://i.sstatic.net/nTIrC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/nTIrC.png" alt="training loss curve(green) and validation loss curve(gray)" /></a></p>
  <p>The green curve is training loss and the gray one is validation loss. I know that before epoch 394 the model in heavily overfitted, but I have no idea what happened after that.</p>
  <p>Also, this is accuracy curves if it helps:</p>
  <p><a href="https://i.sstatic.net/u7R6t.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/u7R6t.png" alt="accuracy curves" /></a></p>
  <p>I'm using categorical cross-entropy and this is the model I am using:
  <a href="https://i.sstatic.net/tcPrn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tcPrn.png" alt="model architecture" /></a></p>
  <p>and here is link to PhysioNet's challenge which I am working on: <a href="https://physionet.org/content/challenge-2017/1.0.0/" rel="nofollow noreferrer">https://physionet.org/content/challenge-2017/1.0.0/</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-14T10:22:57.190Z
Comments.Text: Maybe you should provide a little bit of context, i.e. which task you are trying to solve with CNNs (i.e. which dataset), which loss function you are using, maybe you should do `plot_model(your_model)` and report the architecture of your model.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-14T10:35:43.080Z
Comments.Text: I'm trying to classify ECG signals in PhysioNet 2017 challenge. My output is four classes and I'm using categorical cross-entropy as loss function.
Comments.UserDisplayName: Sepehr Golestanian
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-14T10:39:32.050Z
Comments.Text: Please, edit your post to include these details (maybe also a link to the challenge).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-14T10:59:22.243Z
Comments.Text: I've edited the question.
Comments.UserDisplayName: Sepehr Golestanian
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-14T11:08:09.817Z
Comments.Text: Why are you using a CNN to classify ECG signals? Aren't ECG signals just numerical time series?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-14T11:15:02.237Z
Comments.Text: Yes but I'm not using them as input to my model, I'm using some time-frequency features.
Comments.UserDisplayName: Sepehr Golestanian
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-14T22:11:10.937Z
Comments.Text: Ok. Please, edit your post to clarify that your data is ECG but you're actually pre-processing it first and feeding actually time-frequency features, if I understood correctly. I don't know if this will really be useful to solve your problem, but it might. Anyway, why did you use convolutional layers?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-15T07:37:30.453Z
Comments.Text: Because frequency-time features are images, and as you know convs are useful in image classification. By the way,I didn't mentioned that there's an article I'm trying to implement and this is the network used in article.
Comments.UserDisplayName: Sepehr Golestanian
Tags: <convolutional-neural-networks><training><overfitting><loss>
Title: Why do the training and validation loss curves diverge?
UserDisplayName: Sepehr Golestanian
ThreadText: # Why do the training and validation loss curves diverge?
  
  (Asked by: username_0 on 2020-12-14)
  
  I was training a CNN model on TensorFlow. After a while I came back and saw this loss curve:
  
  [![training loss curve(green) and validation loss curve(gray)](https://i.sstatic.net/nTIrC.png)](https://i.sstatic.net/nTIrC.png)
  
  The green curve is training loss and the gray one is validation loss. I know that before epoch 394 the model in heavily overfitted, but I have no idea what happened after that.
  
  Also, this is accuracy curves if it helps:
  
  [![accuracy curves](https://i.sstatic.net/u7R6t.png)](https://i.sstatic.net/u7R6t.png)
  
  I'm using categorical cross-entropy and this is the model I am using:
  [![model architecture](https://i.sstatic.net/tcPrn.png)](https://i.sstatic.net/tcPrn.png)
  
  and here is link to PhysioNet's challenge which I am working on: <https://physionet.org/content/challenge-2017/1.0.0/>
  
  **Tags:** convolutional-neural-networks, training, overfitting, loss
  
  > **Comments:**
  > **username_1**: Maybe you should provide a little bit of context, i.e. which task you are trying to solve with CNNs (i.e. which dataset), which loss function you are using, maybe you should do `plot_model(your_model)` and report the architecture of your model.  
  > **username_0**: I'm trying to classify ECG signals in PhysioNet 2017 challenge. My output is four classes and I'm using categorical cross-entropy as loss function.  
  > **username_1**: Please, edit your post to include these details (maybe also a link to the challenge).  
  > **username_0**: I've edited the question.  
  > **username_1**: Why are you using a CNN to classify ECG signals? Aren't ECG signals just numerical time series?  
  > **username_0**: Yes but I'm not using them as input to my model, I'm using some time-frequency features.  
  > **username_1**: Ok. Please, edit your post to clarify that your data is ECG but you're actually pre-processing it first and feeding actually time-frequency features, if I understood correctly. I don't know if this will really be useful to solve your problem, but it might. Anyway, why did you use convolutional layers?  
  > **username_0**: Because frequency-time features are images, and as you know convs are useful in image classification. By the way,I didn't mentioned that there's an article I'm trying to implement and this is the network used in article.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-12-14. Score: 0
  
  Follow [this answer](https://stackoverflow.com/questions/37044600/sudden-drop-in-accuracy-while-training-a-deep-neural-net) from StackOverflow, I think your problem related to the second cases where your loss gets Nan value.
  
  Maybe you should try to use the larger datatype (for example float 16 -> float 32)
  
  > **Comments:**
  > **username_0**: As I'm using tensorflow's built-in categorical cross-entropy, I think tf's team knows they should take good care of NaN, values. By the way, can regularization help in this situation or should I put weight constraints?  
  > **username_2**: We have to dig deeper into TF code to get the final conclusion. For the second question, I think yes, regularization is to solve the overfitting problem, here, before epoch 394, your validation accuracy is stable so I can say your model get overfit problem  
  > **username_0**: Thanks very much, I'll try this and update the question!  
  > **username_1**: @username_0 So, why did you accept this answer? What was the problem in the end?  
  > **username_2**: @username_1, you look like the admin of this community, I saw you carefully edit nearly all the post and ask in very details :smile:  
  > **username_1**: @Toby I am a [moderator](https://meta.stackexchange.com/q/75189/287113). I try to ask for clarification and details to make this site a better and more reliable place.  
  > **username_2**: @username_1 I see, glad this community has someone like you  
  > **username_0**: @Toby exactly! And I'm glad I found this community that has people like him in it!  
  > **username_2**: @username_0 but back to his question, can my answer solve your problem?  
  > **username_0**: @Toby You mean the NaN thing or overfitting problem?  
  > **username_2**: I don't know, which one solves your problem :))  
  
  ---
  
  ### Answer by username_3 on 2023-03-11. Score: 0
  
  You have to keep the training loss value constant. Beyond that, your teaching technique or network model is not suitable for the job. I recommend that you are using MLP.
  
  
  ---
  
  