Yes, your assumption is correct as the performance measures in samples/sec should increase in the optimal case and the epoch time should thus be lower, as seen in e.g. Efficientnet-b0 1.
To further isolate the bottlenecks, you could profile the code with e.g Nsight Systems.