I am newer to ML so take what I say with a grain
Loading the data is what I think is the slowest part.  I notice that between epochs my CPU is always 100-200% even thought my device prints as Device: Tesla P100-PCIE-16GB. Then, during a short portion of every epoch my GPU also goes to 100%.
Subscribed to see what experts have to say.