Sorry for the late reply. Since this is the only module that is taking a lot of time execution time, led me to think that data loading could be a problem as also I am using batch size of 1.  I tried to run my whole program in the docker with GPU. Except for the rest of the code, this module takes hours to execute with 400K records. Apart from this module I have not made any data transfer to cuda in my program from dataloaders. The Gpu utilization shows only 4-5% GPU usage.
Kindly suggest how to pre load the entire dataset to GPU as I have run the same program in Google colab pro multiple times but still facing the same issue.
Link to the program in google colab:
Google Colab.
@ptrblck : Please have a look to my code if possible.