How is your training accuracy? Could you overfit your data using 4000 images?
If not, could you first try to get a nearly perfect accuracy using very few samples, e.g. 10?
If that’s not possible, you might have a bug somewhere, e.g. forgetting to zero out the gradients.
How are you normalizing your images? Are you using the ImageNet mean and std or are you calculating both using your data?
What kind of data are you using, i.e. did you generate it yourself or are you using a dataset from someone else? Can you estimate how clean the data is? Could it be that expression labels are mixed sometimes?
How large is your batch size? A small batch size will look noisier than a bigger one.
If your loss does not have a decreasing trend, the training is stuck.
Could you post your code so that we can have a look for obvious bugs?