I tried what you suggest with the simpler convolutional network since that was where the total time difference appeared. Both data[0] and item() accumulate the losses correctly no matter what order they are in.
This is when .data[0] is before .item()
Cycle:0
total time 16.52 loading time 11.07, cuda transfer time 0.11, forward time: 1.40, cost time 0.02, step time 3.90, average cost time with .data[0] 0.01, average cost time with .item() 0.01
cost with data[0] is 3.51052 cost with .item() is 3.51052

Cycle:  1
total time 16.56 loading time 11.10, cuda transfer time 0.11, forward time: 1.40, cost time 0.02, step time 3.90, average cost time with .data[0] 0.01, average cost time with .item() 0.01
cost with data[0] is 2.21125 cost with .item() is 2.21125

Cycle:  2
total time 16.61 loading time 11.17, cuda transfer time 0.11, forward time: 1.39, cost time 0.02, step time 3.89, average cost time with .data[0] 0.01, average cost time with .item() 0.01
cost with data[0] is 2.01861 cost with .item() is 2.01861

This is when .item[0] is before .data[0]
Cycle:0
total time 16.52 loading time 11.08, cuda transfer time 0.11, forward time: 1.40, cost time 0.02, step time 3.89, average cost time with .data[0] 0.01, average cost time with .item() 0.01
cost with data[0] is 3.51901 cost with .item() is 3.51901

Cycle:  1
total time 16.51 loading time 11.06, cuda transfer time 0.11, forward time: 1.40, cost time 0.02, step time 3.90, average cost time with .data[0] 0.01, average cost time with .item() 0.01
cost with data[0] is 2.23629 cost with .item() is 2.23629

Cycle: 3 
total time 16.47 loading time 11.03, cuda transfer time 0.11, forward time: 1.40, cost time 0.02, step time 3.90, average cost time with .data[0] 0.01, average cost time with .item() 0.01
cost with data[0] is 2.05838 cost with .item() is 2.05838

This is when I only use .data[0]
Cycle:  0
total time 16.55 loading time 11.10, cuda transfer time 0.11, forward time: 1.40, cost time 0.02, step time 3.90, average cost time with data 0.01
cost with data[0] is 3.47288

Cycle:  1
total time 16.54 loading time 11.10, cuda transfer time 0.11, forward time: 1.40, cost time 0.02, step time 3.90, average cost time with data 0.01
cost with data[0] is 2.18770

Note that in the previous attempts when I did not use sync with only .data[0] it was
Cycle 0
total time 12.11 loading time 10.80, cuda transfer time 0.11, forward time: 0.38, cost time 0.02, step time 0.80, average cost time 0.01
Cycle:  1
total time 12.05 loading time 10.75, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.80, average cost time 0.01


Does that mean when you use .data[0] there is some synchronization that actually never happens unless you specifically ask for manual sync. This behaviour only happens in smaller networks so this might be also related to the amount of mem available on GPU