Copying weights from one net to another
Is there a canonical method to copy weights from one network to another of identical structure?
rbrigden
<SCODE>net1.load_state_dict(net2.state_dict())
<ECODE>
fmassa
How does deep copy / canonical copy differ from normal weights loading?
FuriouslyCurious
it avoids you from having to do <SCODE>model1 = Model()
model2 = Model()

model2.load_state_dict(model1.state_dict())
<ECODE> and instead you only do <SCODE>model1 = Model()
model2 = copy.deepcopy(model1)
<ECODE>
fmassa
As far as I have seen the code “load_state_dict copies only parameters and buffers”. Does deepcopy also copies only _parameters and _buffers or the hooks as well?
shubhamjain0594
deep copy will recursively copy every member of an object, so it copies everything
qq456cvb
If using deepcopy in my experience, optimizer does not work …
spnova12
Example: weights_new = k*weights_old + (1-k)*weights_new How can I do this?
Navneet_M_Kumar
bhushans23
Hi, have you found a effective way to do this thing?
D-X-Y
I am doing this for each layer. I believe there must be a better method but this works for now.
Navneet_M_Kumar
Navneet_M_Kumar
I think you need to reinitialize the optimizer using the new copied model and then you can copy the optimizer inner values from one to the other, a bit of a mess. I would probably stop at reinitializing the optimizer.
roee
bhushans23
Less messy than what I assumed, good to know! thx!
roee
does one know to to properly solve the polykia averaging issue mentioned? The solution mentioned doesn’t work I believe:
Brando_Miranda
<SCODE>def polyak_update(polyak_factor, target_network, network):
    for target_param, param in zip(target_network.parameters(), network.parameters()):
        target_param.data.copy_(polyak_factor*param.data + target_param.data*(1.0 - polyak_factor))
<ECODE>
Navneet_M_Kumar
load_state_dict worked for me. copy.deepcopy failed for me with run time error “Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment”
Krishna_Garg