If you add other layers to your model, you would have to set strict=False while loading the state_dict to avoid missing keys errors.
The skip connections might just be added in your forward method, which should work fine.