Ah, thanks. I’ve misunderstood the timings, but saw a minor speedup on an A100 using your benchmark.
In any case, it seems the workload is heavily CPU bound. Also there are some unexpected device_to_device kernels, so I’ll look into it a bit more.