@ptrblck Could I ask more questions about replacing the layer?
Let me assume a simple model:
import torch
import torch.nn as nn


class Foo(nn.Module):
    def __init__(self):
        super(Foo, self).__init__()
        self.bar = nn.Conv2d(1, 2, 3)
        
foo = Foo()
foo.bar = nn.Linear(4, 5)  # replacement



When replacing the foo.bar layer using assignment, do we need the guard of torch.no_grad() ? I’m concerned it as the new parameters are coming and the old ones are gone. Or we don’t need it as it is just a replacement of the namespace (foo.bar.__dict__)? When replacing the parameter itself, we needed torch.no_grad() as you mentioned in here: How to assign an arbitrary tensor to model's parameter? - #9 by ptrblck.


Are the parameters inside the old layer (here the convolutional layer) freed from memory (because they can no longer be pointed to)?