It likely depends on if your model has enough computation to fully utilize the GPU or if there is a bottleneck somewhere else in the system. If for example the model or the batch size is too small to fully utilize the GPU, then you can see cases where performance really depends on factors like the peak clock rate attained by each setup. What is the utilization reported by nvidia-smi?