The issue was resolved. The was a small change in the FC layer as my initial code ran for 256x256 but in 512x512 the output tensors was very huge nearly 4times the previous model. I reduced the output features in nn.Linear by factor of 10000.