The GPU utilization doesn’t correspond to the speed, so did you profile the code and see a speedup or slowdown?
E.g. if mixed-precision training is giving you a speedup e.g. by using TensorCores, your code might now suffer (more) from a potential data loading bottleneck, which would reduce the GPU utilization.