Hi @ptrblck ,
I tried now to shuffle the data but I figured out why the final loss coming out of the CNN is different when I apply the shuffling to the data as I showed above. I find that even the very first few samples in my batch are not shuffled, however, the floats are not exact in the loss and that is why the final loss and the weights will be updated differently.
So here is the loss when I have no shuffle applied
[214.9400634765625, 205.25238037109375, 204.9016571044922, 207.81027221679688, 204.00399780273438, 203.37063598632812]

And here is how it looks like when I apply shuffle
[214.94007873535156, 205.2523956298828, 204.90162658691406, 207.81028747558594, 204.0040740966797, 203.37075805664062]

You can notice that the decimal points are different which will result in a smooth increase in the final loss but epoch after epoch they will accumulate and accordingly the final accuracy will differ.
Do you have any clue why this occurs?
Also, a weird behavior when I use torch.round on the losses. So the losses in the case where I apply no shuffle should be:
[214.9400634765625, 205.25238037109375, 204.9016571044922, 207.81027221679688, 204.00399780273438, 203.37063598632812]

When I apply rounding the output come out to be:
[215, 208, 211, 216, 212, 216, 209, 225, 207, 212, 220]

You can see they are completely different from their float representation. I do not know what is going on   However, in this case shuffeling and not shuffeling will be the same 
Really thanks and looking forward to your response.