Simpler models don’t seem to exhibit this behavior. For example, this code fully deallocates the memory once all the references are deleted:
import torch
import torch.nn as nn
import gc

class Bloat(nn.Module):
    def __init__(self, size):
        super().__init__()
        self.fc = nn.Linear(size,size)
    def forward(self,x):
        return self.fc(x)

pile = []
for i in range(10):
    pile.append(Bloat(2**12))

del pile
gc.collect()