Thanks, Frank! 
While preparing an example for you, I found some possible reasons for having different losses.

Brief context: I removed the nn.BatchNorm2d() layers to fine-tune a pretrained VGG19 but kept the nn.Dropout(p=0.5) layers (is this strategy fine for fine-tuning?).

At first, I thought the presence of these dropout layers following the nn.Linear layers altered the output every time hence I got different losses. So, I have to perform another experiment just to verify this. Following is what was done next

̶R̶e̶m̶o̶v̶e̶d̶ ̶B̶a̶t̶c̶h̶N̶o̶r̶m̶(̶)̶ ̶l̶a̶y̶e̶r̶s̶
Removed all nn.Dropout() layers
Set a constant seed value for torch/Cuda random number generator
Initialized the weight/bias of all three linear layers (in the classifier part)


To my surprise, the network still generated different outputs for the same inputs for batch size = 1, 2, and 4. The results are tabularized below. Can you please share the needed wisdom to understand this?
image691×332 13.4 KB
The green color indicates the same output for the relevant inputs.
Now, coming back to an older question.



 stark:

Which of these should be given more importance?
a) loss calculated per image (for batch size =1)
b) loss calculated per batch (for batch size > 1)
c) mean loss per epoch (irrespective of the batch size)