Sorry for a late reply… So I checked  all my CPU, GPU and Memory/swap utilization. None of them are used significantly and GPU is only used about 3 GB out of 12 GB and RAM utilization is low too. I checked the variable types inside the forward function using  [var].is_cuda and [var].type … both of them show that they are definitely Cuda variables.