I have similar problem which is as follows:
I have a Unet structure which I try to feed back some feature maps, and I am using multi-gpu
when I try to run my code with only one gpu core it throws this error
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
but when I run it with two cores it goes fin.
here how I use the backward
curr_loss = self.criterion(smax_outputs, label)
    if self.gpustat['n_gpus'] >= 2:
      Dev_Inc = max(self.cfg['TRAIN_BATCH_SIZE'] // self.gpustat['n_gpus'], 1)
      idx = torch.ones(Dev_Inc).cuda()
      curr_loss.backward(idx)
    else:
      curr_loss.backward()
    
    self.optimizer.step()
    # zero the parameter gradients
    self.optimizer.zero_grad()

it is not a problem of memory…