Is there anyway to let pytorch reserve less GPU memory? I found it is reserving GPU memory very aggressively even for simple computation, which causes CUDA OOM for large computations. Here is some code snippet
In [1]: import torch as tc
In [2]: m = tc.nn.Sequential(
   ...:     tc.nn.Linear(1000, 1000),
   ...:     tc.nn.Linear(1000, 1000),
   ...:     tc.nn.Linear(1000, 1000),
   ...:     tc.nn.Linear(1000, 1),
   ...: ).cuda()
In [3]: x = tc.randn(1000, 1000).cuda()
In [4]: m(x).sum().backward()  # a forward pass followed by a backward pass

This causes GPU memory usage of 1285MB from nvidia-smi
|    0   N/A  N/A     31758	 C   ...-3.7.6/bin/python3.7     1285MiB |

But there are only very few tensors allocated on gpu
In [10]: import gc

In [11]: [obj.shape for obj in gc.get_objects() if tc.is_tensor(obj) and obj.device.type != 'cpu']
Out[11]:
[torch.Size([1000, 1000]),
 torch.Size([1000, 1000]),
 torch.Size([1000]),
 torch.Size([1000, 1000]),
 torch.Size([1000]),
 torch.Size([1000, 1000]),
 torch.Size([1000]),
 torch.Size([1, 1000]),
 torch.Size([1])]

I don’t want to call torch.cuda.empty_cache() since it is super expensive in training loop.
Please advice, thank you!