The memory management docs 3 would give you some more information.
I’m not able to reproduce the same memory usage and see the expected reduction by 2 using float16 for a single tensor:
x = torch.randn(1024, 1024, dtype=torch.float32, device='cuda') # switch the dtype to torch.float16

Output:
# float16:
|---------------------------------------------------------------------------|
| Allocated memory      |    2048 KB |    2048 KB |    2048 KB |       0 B  |
|       from large pool |    2048 KB |    2048 KB |    2048 KB |       0 B  |
|       from small pool |       0 KB |       0 KB |       0 KB |       0 B  |
|---------------------------------------------------------------------------|
# float32:
|---------------------------------------------------------------------------|
| Allocated memory      |    4096 KB |    4096 KB |    4096 KB |       0 B  |
|       from large pool |    4096 KB |    4096 KB |    4096 KB |       0 B  |
|       from small pool |       0 KB |       0 KB |       0 KB |       0 B  |
|---------------------------------------------------------------------------|

as well as torchvision.models.resnet18:
# float16:
|---------------------------------------------------------------------------|
| Allocated memory      |   23768 KB |   23768 KB |   23768 KB |       0 B  |
|       from large pool |   20480 KB |   20480 KB |   20480 KB |       0 B  |
|       from small pool |    3288 KB |    3288 KB |    3288 KB |       0 B  |
|---------------------------------------------------------------------------|
# float32:
|---------------------------------------------------------------------------|
| Allocated memory      |   45763 KB |   45763 KB |   45763 KB |       0 B  |
|       from large pool |   42368 KB |   42368 KB |   42368 KB |       0 B  |
|       from small pool |    3395 KB |    3395 KB |    3395 KB |       0 B  |
|---------------------------------------------------------------------------|