Thank you for your reply 
Are you suggesting the rest of the allocation is CUDA libraries (which should then take constant space)?
It doesn’t seem to be the case when I increase the input size
In [1]: import torch as tc
In [2]: m = tc.nn.Sequential(
   ...:     tc.nn.Linear(1000, 1000),
   ...:     tc.nn.Linear(1000, 1000),
   ...:     tc.nn.Linear(1000, 1000),
   ...:     tc.nn.Linear(1000, 1),
   ...: ).cuda()
In [3]: x = tc.randn(100000, 1000).cuda()  # 400MB

In [7]: m(x).sum().backward()

In [8]: tc.cuda.memory_allocated()  # 1600MB, this number makes sense to me
Out[8]: 1627630080

In [9]: tc.cuda.memory_reserved()  # why double the above size?
Out[9]: 3338665984

And after this call, nvidia-smi shows even bigger memory footprint
|===============================+======================+======================|
|   0  A100-SXM4-40GB	   Off  | 00000000:10:1C.0 Off |                    0 |
| N/A   40C    P0    70W / 350W |   4430MiB / 40537MiB |      0%      Default |
|                               |                      |             Disabled |

IIUC, 4 CUDA tensors of 400MB should be created (one for input, 3 for intermediate result right after each linear layer for gradient computation). And the rest should be negligible. So the total CUDA memory footprint should be 1.6GB. But tc.cuda.memory_reserved() shows 200% as much, and nvidia-smi uses even more.
Any idea?
thanks!