Thanks a lot for pointing that out.
I face this error when not using any pretrained model. This happens with random weights (training from scratch) (Actually I never use the load_my_state_dict for loading the weight. I should have removed it earlier to prevent any confusion)
For the initialization part, is this alright to do ? :
def init_weights(m):
    if type(m) == nn.Conv2d:
        nn.init.xavier_uniform_(m.weight.data, gain=nn.init.calculate_gain('relu'))
        
model.apply(init_weights)


or should this simply does it:
for m in model.modules():
    if isinstance(m, nn.Conv2d):
        nn.init.xavier_uniform_(m.weight.data, gain=nn.init.calculate_gain('relu'))
return model