@mrshenli okay thanks I’ll try this. But also this means all of my other timings have been wrong. So thanks for showing me how to use cuda events too.
edit:
@mrshenli it turns out you need a synchronize after all
        start = time.monotonic()
        s = torch.cuda.current_stream(rank)
        e_start = torch.cuda.Event(enable_timing=True)
        e_finish = torch.cuda.Event(enable_timing=True)
        s.record_event(e_start)
        for i in range(10):
            img_tensor = img_tensor_cpu.to(rank)
            # torch.cuda.synchronize(rank)
            dogs.append(dog(img_tensor))
        torch.cuda.synchronize(rank)
        s.record_event(e_finish)
        e_finish.synchronize()
        end = time.monotonic()

gives me this
====== size = 1  ======

rank 0 Iteration 9 forward latency is 1283.0596923828125
end - start =  1.2832060940563679


====== size = 2  ======

rank 0 Iteration 9 forward latency is 626.5835571289062
end - start =  0.6267032357864082
rank 1 Iteration 9 forward latency is 640.3717041015625
end - start =  0.6404897100292146


====== size = 3  ======

rank 0 Iteration 9 forward latency is 443.1278076171875
end - start =  0.44322703895159066
rank 1 Iteration 9 forward latency is 471.8766174316406
end - start =  0.47198665188625455
rank 2 Iteration 9 forward latency is 461.29559326171875
end - start =  0.46140363393351436


====== size = 4  ======

rank 0 Iteration 9 forward latency is 397.9264221191406
end - start =  0.3981346560176462
rank 2 Iteration 9 forward latency is 374.9112243652344
end - start =  0.3749916541855782
rank 3 Iteration 9 forward latency is 360.9978942871094
end - start =  0.3610941809602082
rank 1 Iteration 9 forward latency is 362.57073974609375
end - start =  0.3626508240122348