If you are timing CUDA operations you would have to synchronize the code before starting and stopping the timer due to the asynchronous execution of CUDA kernels via torch.cuda.synchronize().



 mhong94:

The data is stored on my local SSD, and I’m not exactly sure where on the network it’s stored.


I don’t understand this explanation. Is the data stored on the SSD in your workstation or on a server in your network (or both)?
In the latter case you would introduce the network latency into the training, so I would recommend to store the data on a local SSD.



 mhong94:

The original code used ImageFolder to load the data, and I tried changing it to a standard dataset, but this did not help


What do you mean by “standard Dataset”? Did you write a custom Dataset?