Thank you @ptrblck for your response, yes I did as shown in the first post.
When I run this code, and I add manually tensors to cuda, with ponctual operations, I can see GPU being consumed:
import torch
a = torch.rand(20000,20000).cuda()
while True:
    a += 1
    a -= 1
    print("Allocated:", round(torch.cuda.memory_allocated(0)/1024**3,1), "GB")

It raises the % of utilisation of GPU from 1.7GB to 3.2GB, as shown in the nvidia-smi command results, please see the nvidia-smi screenshots of my first post.
I’ll try with the packages you listed, thank you.
What might be the solutions to solve this GPU Bottleneck ? I can try them all.
Thank you very much