This seems like it’s related to the way you saved the model, rather than the changes in the architecture.
Instead of doing model.load_state_dict(net), try model.load_state_dict(net.state_dict)! The state_dict of the model is the weights and biases of the network, whereas the model itself is the entire object, with the architecture, weights and other elements included.