Hi!
Following your advice, the situation changed. Please look at the following picture, which summarizes the average computation times of the train function:

Screenshot 2020-03-15 at 15.06.242070×832 210 KB

It turns out that the loss computation indeed can be ignored. Regarding the model + sync:
*) When in DP mode, on both devices model.forward(...) is entered and left at nearly identical ms timestamps. The difference is max 5 ms (which is ok I guess). The average time diff between entering and leaving model.forward(...) is constant at ~20-40 ms depending on using DP or not.
Is the conclusion correct that the problem is, that the cudas are out of sync?
Since the batches contain data of different lengths, could this be the origin of the problem?
Greetings
Thomas
edit: @ptrblck this was supposed to be an answer to you - missed the reply button