Thank you for your explanation.
I have replaced the line
self.register_buffer("test", torch.Tensor([1,2,3]).view(-1, 1, 1), False)

to
self.register_buffer("test", torch.rand([10000, 10000, 10]), False)

so that I allocate large GPU memory.
The results looks as follows:
< before moving the model to gpu >
buffer test is on cpu
--- nvidia-smi ---
------------------

< after moving the model to gpu >
--- nvidia-smi ---
|    0   N/A  N/A      3401      C   ...rtualenvs/nova/bin/python     4819MiB |
------------------

< after moving the model back to cpu >
--- nvidia-smi ---
|    0   N/A  N/A      3401      C   ...rtualenvs/nova/bin/python     4819MiB |
------------------

< after deleting the model >
--- nvidia-smi ---
|    0   N/A  N/A      3401      C   ...rtualenvs/nova/bin/python     1003MiB |
------------------

The memory 1003MiB seems unrelated to the model as you mention and it’s not being released properly even though I am allocating greater amount of memory