thanks for your answer! And thanks for the fix on torch.cuda.synchronize()   by adding it, the situation becomes:
V0 Total: 2.5034 Loss forward: 0.0111 Loss backward: 1.7371
V0 Total: 0.2608 Loss forward: 0.0004 Loss backward: 0.2297
V0 Total: 0.2657 Loss forward: 0.0061 Loss backward: 0.2203
V0 Total: 0.2759 Loss forward: 0.0068 Loss backward: 0.2181
V0 Total: 0.2736 Loss forward: 0.0067 Loss backward: 0.2147
V0 Total: 0.2715 Loss forward: 0.0047 Loss backward: 0.2148
V0 Total: 0.2697 Loss forward: 0.0040 Loss backward: 0.2150
V0 Total: 0.2614 Loss forward: 0.0001 Loss backward: 0.2159

V1 Total: 1.3935 Loss forward: 0.0006 Loss backward: 0.8544
V1 Total: 0.0790 Loss forward: 0.0041 Loss backward: 0.0590
V1 Total: 0.0854 Loss forward: 0.0068 Loss backward: 0.0594
V1 Total: 0.0631 Loss forward: 0.0059 Loss backward: 0.0390
V1 Total: 0.0669 Loss forward: 0.0048 Loss backward: 0.0536
V1 Total: 0.0844 Loss forward: 0.0068 Loss backward: 0.0584
V1 Total: 0.0596 Loss forward: 0.0040 Loss backward: 0.0368
V1 Total: 0.0669 Loss forward: 0.0026 Loss backward: 0.0560

The difference in the forward has disappeared. So do I understand correctly that by calling the synchronize we cancel any influence of the queuing?
The difference in the backward is still a bit puzzling, since the V0 model has 2.3x more operations (back of the envelope calculation), but the ratio here is more like 1/4.