I had actually missed the nvidia-smi output:
It shows that 2779MiB / 6144MiB (46%) is being utilzed. This implies that the model was successfully loaded into the GPU. One empirical way to verify this is to time it using device = 'cpu' and then time it using device = 'cuda' and verify the different runtimes for a batch size greater than 1(Preferabbly, keep as high a batch size as possible). If the runtimes are the same, there is indeed some issue. Really do not have an answer for the 2% part, though.