Thank you I am currently looking at a progressive gan implementation in pytorch and just like you said they would re-instantiate their network, copy and paste back the state_dict every time when they want to grow their network.  It looks easy enough to implement I might just do that first.