I am doing 2 class classification with CNN for unbalanced data (both training and test). As go through the training, I see the change in the loss of the valid set. The accuracy of the valid set is very volatile. The best accuracy point of the valid set is stored, but the loss at the moment of being stored is rather bigger than the other. I thought it might be overfitting, so I applied dropout or batch norm, but there was no big change. loss and acc change as follows: (Above: training, below: valid)
Epoch: 0
| Loss: 0.749 | Acc: 55.214% (27607/50000)
| Loss: 0.690 | Acc: 54.150% (5415/10000)
Saving …
Epoch: 1
| Loss: 0.690 | Acc: 56.218% (28109/50000)
| Loss: 0.699 | Acc: 67.870% (6787/10000)
Saving …
Epoch: 2
| Loss: 0.690 | Acc: 56.452% (28226/50000)
| Loss: 0.688 | Acc: 50.810% (5081/10000)
…
Epoch: 67
| Loss: 0.547 | Acc: 72.024% (36012/50000)
| Loss: 0.796 | Acc: 62.190% (6219/10000)
Epoch: 68
| Loss: 0.537 | Acc: 72.214% (36107/50000)
| Loss: 0.828 | Acc: 68.090% (6809/10000)
Saving …
…
Epoch: 94
| Loss: 0.455 | Acc: 77.986% (38993/50000)
| Loss: 1.016 | Acc: 69.190% (6919/10000)
Saving …
Epoch: 95
| Loss: 0.451 | Acc: 78.082% (39041/50000)
| Loss: 0.994 | Acc: 69.010% (6901/10000)
Epoch: 96
| Loss: 0.450 | Acc: 78.114% (39057/50000)
| Loss: 0.955 | Acc: 67.390% (6739/10000)
Epoch: 97
| Loss: 0.444 | Acc: 78.386% (39193/50000)
| Loss: 0.996 | Acc: 68.520% (6852/10000)
Epoch: 98
| Loss: 0.441 | Acc: 78.844% (39422/50000)
| Loss: 1.000 | Acc: 67.530% (6753/10000)
Epoch: 99
| Loss: 0.441 | Acc: 78.618% (39309/50000)
| Loss: 0.934 | Acc: 66.080% (6608/10000)
In this case, how can I reduce the loss of the valid set? Is there a case in which training does not apply well to valid set because of data set property?