I found out that deleting tensor.data is impossible.
My test code:
a = torch.zeros([32, 64], requires_grad=True).cuda()
b = torch.zeros([64, 128], requires_grad=True).cuda()
c = torch.zeros([128, 256], requires_grad=True).cuda()
print("Calculate result four times: ", torch.cuda.memory_allocated())
d = torch.matmul(a, b)
e = torch.matmul(d, c)
print("Calculate result four times: ", torch.cuda.memory_allocated())
del d.data
print("Calculate result four times: ", torch.cuda.memory_allocated())
e.mean().backward()
print("Calculate result four times: ", torch.cuda.memory_allocated())
print(a.grad)
print(b.grad)