I found an unrelated bug:
model = nn.Sequential(*layers)
print(model)
for m in self.modules():
    if isinstance(m, nn.Conv2d):
        nn.init.xavier_uniform_(m.weight.data, gain=nn.init.calculate_gain('relu'))

Here, self.modules() doesn’t have your model yet, so you are not initializing with xavier uniform on the layers in model.
About your original question, this seems strange, and looks like it should not happen, your model definition looks fine.
I suspect that your load_my_state_dict is doing something funky, though I cant nail it yet.
If you dont load from a state_dict and start your model with random weights, does the error still happen?