One silly reason (which has happened to me) could be because of the transforms in your loader. If the loader has shuffle = True, or something like RandomHorizontalCrop() then the input to the network is changing every time you run your code, so naturally, the output changes slightly every time.