Answers.Body: <p>If your accuracy is 50% in train and val, would you say that the model fit the data perfectly?</p>
  <p>One guess is that your val and train data are pretty similar. Have you split the data before training the model?</p>
  <p>The loss seems quite high. What does your accuracy look like? Have you tried to train for more epochs?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T11:06:43.563Z
Answers.Text: Accuracy on both data are around 95% at the end of the learning. Yes I have split the data. Yes I have tried to train for more epochs and the losses still seem to be identical.
Answers.UserDisplayName: tail
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T11:06:49.787Z
Answers.Text: 50% accuracy is relative.  On a 10 class problem, 50% is much better than on a 2 class problem.  Also, loss is entirely relative and mostly meaningless without another loss for the same problem (and same loss function) to compare it to.  Here we have only the validation loss for comparison.
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T11:09:20.823Z
Answers.Text: @tail
  Have you tried kfold split? https://www.askpython.com/python/examples/k-fold-cross-validation
Answers.UserDisplayName: Skobo Do
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T11:13:07.410Z
Answers.Text: @DavidHoelzer that's the point. It's relative. It does highly depend on the data chosen for training and validation. I can have easily 100% accuracy on a multiclass classification with arbitrary number of classes when choosing the validation data-set accordingly.
Answers.UserDisplayName: Skobo Do
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T11:16:44.537Z
Answers.Text: @DavidHoelzer
  "50% accuracy is relative. On a 10 class problem, 50% is much better than on a 2 class problem."
  That not generally true. It depends on whether your data is balanced or not. For balanced data, it would be true.
Answers.UserDisplayName: Skobo Do
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-03T11:02:13.123Z
Answers.LastActivityDate: 2022-11-03T11:02:13.123Z
Answers.UserDisplayName: Skobo Do
Body: <p>I am developing an ANN from scratch which classifies MNIST digits.</p>
  <p>These are the curves I get using only one hidden layer composed of <code>100</code> neurons activated by <code>ReLU</code> function. The output's neurons are activated by the <code>softmax</code> function:</p>
  <p><a href="https://i.sstatic.net/m6ttB.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/m6ttB.png" alt="curves" /></a></p>
  <p>Is it correct that training and validation loss are almost identical? Does it mean that my model <em>perfectly fit</em> the data?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-03T10:38:26.803Z
Comments.Text: Usually it means you did something wrong. Double check everything.
Comments.UserDisplayName: lpounng
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-03T10:42:49.533Z
Comments.Text: @lpounng What If I do not spot errors?
Comments.UserDisplayName: tail
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-03T11:07:50.313Z
Comments.Text: If you have implemented from scratch, I would verify that your validation data is being held out during training.  It can happen that the two track quite tightly (as these do) but in my experience it is somewhat unusual for them to reflect the same hills and valleys throughout.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-03T15:33:12.293Z
Comments.Text: @DavidHoelzer No bugs spotted...
Comments.UserDisplayName: tail
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-04T01:15:05.443Z
Comments.Text: @tail try get someone else to look at it, or post more details here so we can help.
Comments.UserDisplayName: lpounng
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-27T16:06:32.210Z
Comments.Text: MNIST is a very easy dataset so I'd not be that surprised to have a good validation score. Try with Fashion MNIST which is a bit more challenging to see if the correlation remains.
Comments.UserDisplayName: Lelouch
Tags: <neural-networks><loss><mnist>
Title: Training and validation loss are almost the same (perfect fit?)
UserDisplayName: tail
ThreadText: # Training and validation loss are almost the same (perfect fit?)
  
  (Asked by: username_0 on 2022-11-03)
  
  I am developing an ANN from scratch which classifies MNIST digits.
  
  These are the curves I get using only one hidden layer composed of `100` neurons activated by `ReLU` function. The output's neurons are activated by the `softmax` function:
  
  [![curves](https://i.sstatic.net/m6ttB.png)](https://i.sstatic.net/m6ttB.png)
  
  Is it correct that training and validation loss are almost identical? Does it mean that my model *perfectly fit* the data?
  
  **Tags:** neural-networks, loss, mnist
  
  > **Comments:**
  > **username_1**: Usually it means you did something wrong. Double check everything.  
  > **username_0**: @username_1 What If I do not spot errors?  
  > **username_2**: If you have implemented from scratch, I would verify that your validation data is being held out during training.  It can happen that the two track quite tightly (as these do) but in my experience it is somewhat unusual for them to reflect the same hills and valleys throughout.  
  > **username_0**: @username_2 No bugs spotted...  
  > **username_1**: @username_0 try get someone else to look at it, or post more details here so we can help.  
  > **username_3**: MNIST is a very easy dataset so I'd not be that surprised to have a good validation score. Try with Fashion MNIST which is a bit more challenging to see if the correlation remains.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2022-11-03. Score: 0
  
  If your accuracy is 50% in train and val, would you say that the model fit the data perfectly?
  
  One guess is that your val and train data are pretty similar. Have you split the data before training the model?
  
  The loss seems quite high. What does your accuracy look like? Have you tried to train for more epochs?
  
  > **Comments:**
  > **username_0**: Accuracy on both data are around 95% at the end of the learning. Yes I have split the data. Yes I have tried to train for more epochs and the losses still seem to be identical.  
  > **username_2**: 50% accuracy is relative.  On a 10 class problem, 50% is much better than on a 2 class problem.  Also, loss is entirely relative and mostly meaningless without another loss for the same problem (and same loss function) to compare it to.  Here we have only the validation loss for comparison.  
  > **username_4**: @username_0 Have you tried kfold split? https://www.askpython.com/python/examples/k-fold-cross-validation  
  > **username_4**: @username_2 that's the point. It's relative. It does highly depend on the data chosen for training and validation. I can have easily 100% accuracy on a multiclass classification with arbitrary number of classes when choosing the validation data-set accordingly.  
  > **username_4**: @username_2 "50% accuracy is relative. On a 10 class problem, 50% is much better than on a 2 class problem." That not generally true. It depends on whether your data is balanced or not. For balanced data, it would be true.  
  
  ---
  
  