PyTorch definitely won’t free up memory from layers that you defined in __init__ but didn’t use in forward, because the Parameters those layers use will still be in scope. That should explain the behavior you observe with inheritance.