I assume you are using the proper approach of storing the state_dict instead of the complete model.
If that’s the case, you could remove the keys you don’t want to store to disk.
After loading this smaller state_dict from disk again, load it to your model with strict=False:
model = nn.Sequential(
    nn.Linear(1, 1),
    nn.Linear(1, 2))

sd = model.state_dict()

# Delete unnecessary keys
del sd['0.weight']
del sd['0.bias']

# Store
torch.save(sd, 'tmp.pt')

# Load
model = nn.Sequential(
    nn.Linear(1, 1),
    nn.Linear(1, 2))
sd = torch.load('tmp.pt')
model.load_state_dict(sd, strict=False)