Actually I test it based on a resnet architecture.
The whole code would like this:
Class BasicBlock(nn.Module):
    def __init__():
        super(BasicBlock, self).__init__()
        self.conv1 = nn.Conv2d(64, 64, 3)
        self.conv2 = nn.Conv2d(64, 64, 3)
    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(x)
    return x
class WholeModel(nn.Module):
    def __init__():
        super(WholeModel, self).__init__()
        self.block1 = BasicBlock()
        self.block2 = BasicBlock()
    def forward(self, x):
        out = self.block1(x)
        out = self.block2(out)

While runing the “out = self.block1(x)”, it saves x. And after the first conv in block1: “out = self.conv1(x)”. x is useless anymore. But the input x will not be freed until “out = self.block1(x)” is finished. I think it could be freed earlier since it is useless anymore after the conv1 in block1 is finished. I tried to “del x” after the “out = self.conv1(x)”, but it is obviously useless. Do you have any ideas about how to free the  memory just after the “out  = self.conv1(x)”? Thank youx