You meant something like this? After running this code, CUDA memory footprint seen at nvidia-smi becomes 3.2GB
import torch as tc

m = tc.nn.Sequential(
    tc.nn.Linear(1000, 1000),
    tc.nn.Linear(1000, 1000),
    tc.nn.Linear(1000, 1000),
    tc.nn.Linear(1000, 1),
).cuda()

x = tc.randn(100000, 1000).cuda()  # 400MB

m(x).sum().backward()

+-------------------------------------------------------------------------+
| Processes:                                                              |
|  GPU   GI   CI        PID   Type   Process name              GPU Memory |
|        ID   ID                                               Usage      |
|=========================================================================|
|    0   N/A  N/A     79663      C   ...n-3.7.6/bin/python3.7     3281MiB |
+-------------------------------------------------------------------------+