Hi,
I am evaluating my model, some part of the code is like this:
        eval_scales = (0.5, 0.75, 1, 1.25, 1.5, 2)
        eval_flip = True
        for i, (imgs, label) in diter:
            N, _, H, W = label.shape
            probs = torch.zeros((N, n_classes, H, W))
            probs.requires_grad = False
         
            for sc in eval_scales:
                new_hw = [int(H*sc), int(W*sc)]
                with torch.no_grad():
                    im = F.interpolate(imgs, new_hw, mode='bilinear', align_corners=True)
                    im = im.cuda()
                    out = net(im)
                    out = F.interpolate(out, (H, W), mode='bilinear', align_corners=True)
                    prob = F.softmax(out, 1)
                    probs += prob.cpu()
                    if eval_flip:
                        out = net(torch.flip(im, dims=(3,)))
                        out = torch.flip(out, dims=(3,))
                        out = F.interpolate(out, (H, W), mode='bilinear',
                                align_corners=True)
                        prob = F.softmax(out, 1)
                        probs += prob.cpu()
                    del out, prob

Problem is like this: If I use eval_scales=(2,), it takes around 4G of my gpu memory. However, if I use eval_scales=(0.5, 0.75, 1, 1.25, 1.5, 2). The memory usage would reach as much as 8G or so.
I wonder why the memory will not be released in time. Since the memory is actually used 4G at most when I use an input scale of 2, the other smaller scales should not matter and takes extra memory. How could I save these extra memory please ?