Could you check the utilization reported by nvidia-smi? You might want to increase the batch size until you are close to using all available memory or the throughput in examples/sec is no longer improving.