Thank you for the replies.
My train-dataset was shuffled, so the train phase itself should be ok.
The cause of the instability was, as ptrblck indicated, lack of model.eval() and instead I was using torch.no_grad().
Since my model contains batch normalization, it makes sence that inference results were different without model.eval().
Thanks again.