Thanks for the update. To further isolate the bottleneck, you could remove the data loading part and use pre-allocated tensors in the posted batch sizes and profile the code again.
If you are seeing the expected speedup, the bottleneck might be coming from the data loading itself.