If you are working with a small dataset, you could preload the whole dataset and push it to the GPU before training, to avoid loading times. Also, make sure not to create any unnecessary synchronization points in your training loop, e.g. printing the loss often.