Update: I found what I was looking for!
Torch (in version 0.4.0)  takes about 2GB CPU RAM upfront when the first cuda/GPU tensor is allocated. That was not a memory leak.
Line #    Mem usage    Increment   Line Contents
================================================
     4     29.2 MiB     29.2 MiB   @profile
     5                             def main():
     6     95.6 MiB     66.4 MiB       import torch
     7     95.6 MiB      0.0 MiB       size = (1024, 1024, 10)
     8     95.6 MiB      0.0 MiB       cpu, gpu = torch.device('cpu'), torch.device('cuda:0')
     9                                 # couple of CPU tensors
    10    135.6 MiB     40.0 MiB       t_cpu1 = torch.zeros(size, device=cpu)
    11    175.6 MiB     40.0 MiB       t_cpu2 = torch.zeros(size, device=cpu)
    12                                 # couple of GPU tensors
    13   2120.5 MiB   1944.9 MiB       t_gpu1 = torch.zeros(size, device=gpu)
    14   2120.5 MiB      0.0 MiB       t_gpu2 = torch.zeros(size, device=gpu)
    15   2120.6 MiB      0.1 MiB       t_gpu3 = t_cpu1.to(gpu)
    16   2080.6 MiB    -40.0 MiB       del t_cpu1
    17   2080.6 MiB      0.0 MiB       gc.collect()  # does this free memory of t_cpu1 ?
    18   2040.6 MiB    -40.0 MiB       del t_cpu2
    19   2040.6 MiB      0.0 MiB       gc.collect()  # does this free memory of t_cpu2?
    20   2040.6 MiB      0.0 MiB       del t_gpu1, t_gpu2, t_gpu3
    21   2040.6 MiB      0.0 MiB       gc.collect()  # these doesnt take any CPU RAM !!
    22   2040.6 MiB      0.0 MiB       del torch    # remove torch module from context
    23   2040.6 MiB      0.0 MiB       gc.collect()  # does it free everything ?

P.S.
the memory leak was somewhere else on my code, I was misled by the initial 2GB unaccounted memory.
Thanks, to anyone who attempted to answer my question!