okay i removed the sync and enclosed the loop in time.monotonic.
2 gpus:
1.2319540430326015
1.3151386808604002
total: 6.296403981978074

3 gpus:
1.1622967889998108
1.3194731972180307
1.3116707119625062
total: 5.875259702792391

4 gpus:
1.1516663811635226
1.4554521720856428
1.76222850009799
1.8313195349182934
total: 6.504983321996406

i ran this several times it’s consistent.