thank you for the answer!
My question was motivated precisely due to the doubling of memory usage during validation!
Despite wrapping it with
with torch.no_grad()

As soon as I reach validation, my memory usage doubles, then it increases again when I compute the Class Activation Maps…
Could it be caused by me doing:
for input,label in val_loader:
      input = input.to(self.device)
      label = label.to(device)

But as far as I remember, I was getting a
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

error if I didn’t place the inputs and labels on the device…