In my understanding, GPU speed depends on many things:
0. Batch size
If the batch size is less, more time will be spent on data transfer rather than any useful work with GPU.
1. The temperature of the GPU
If the temperature is too much for the GPU to handle, it will enable hardware/software speed throttling.
2. The hard drive speed (whether local drive/network drive)
Whether you are loading from a local SATA / SSD drive or if the data is located in a network drive.
3. The processor speed (and the time it takes to populate the cache)
In my observation, sometimes it takes some time for the processor to get into full utilization and read/populate data in cache. Especially after the system reboot. So, wait for some time and observe if the speed improves.
Check if any of these things help.