The OOM might come from the deepcopy.
How close the the memory limit are you before copying the modules?
Are you able to train the original model, since the forward and backward passes should also use some memory (which should be larger than copying some modules).
Have a look at this post 5 for an example how to derive a custom class from a torchvision model.