tius:

However, when querying the TorchServe for inference, the prediction was done entirely on CPU (as the CPU load was close to 100%, and the gpu was idle at 0% to 2%).


This could point towards a CPU bottleneck, e.g. in the data loading.
Try to profile the code to see which operations are the actual bottleneck and to verify that the GPU is indeed used.