Vincent:

detach will create a new tensor for modules


Then, consider using detach_(), which is an inplace operation.
If you want torch.no_grad() for defrosting the model during training, how about something like this?:
if freeze is True:
    with torch.no_grad():
        xxnet()(input_tensor)
else:
    xxnet()(input_tensor)