I think I know why the memory usage is high. I used cudnn.benchmark = True in my code. This caused a high memory usage. For the code below:
import torch
import torch.backends.cudnn as cudnn
import pdb
from torchvision.models import resnet50

net = resnet50()
net.eval()
# cudnn.benchmark = True
net = net.cuda()

with torch.no_grad():
    for i in range(100):
        img = torch.randn(1, 3, 512, 512).cuda()
        c_pred = net(img).squeeze()
        print(torch.cuda.memory_reserved()/1024/1024)
        print(torch.cuda.max_memory_reserved()/1024/1024)
        pdb.set_trace()

The memory usage is:

And if I uncomment the cudnn.benchmark = True, the memory usage is:

That’s quite too much memory used.  Then I used torch.cuda.empty_cache(), the memory usage became normal.

All the results are tested on one RTX2080.