Your training might suffer from e.g. a data loading bottleneck (or any other bottleneck, which starves the GPU). Have a look at e.g. this post 2 for more information about the data loading and try to profile your code to check, which is the slowest part.