CUDA uses some memory for its context. This is usually done in the first CUDA call.
On my system this takes approx. 400 MB on the GPU.
Have a look at the memory usage before importing pytorch, then call torch.cuda.init() and see the growth.
This means, that the 1GB aren’t used completely by the 5 models.
Could you check this on your machine?