Apart from @ptrblck’s comment, how large is your dataset? For example, if you use a rather small dataset (maybe just a sample for testing the network), then the distribution of the training and test data could be very different. This would yield in the trends of the losses you observe.