The results are below. It seems almost if .item() causes for some synchronization that is not needed when .data[0] is used. Most of the difference is definitely related to the synchronization of calculations that happen at the update step.
When I put sync before .data[0] the total time is about 12 whereas if I put it at the end of the cycle it is 16. If I remove all syncs inside the for loop over the database but put it after the whole loop just before when the final measurement is made the time is again 12.  What does this suggest? That there is some unecessary sync done by the .item() that is not done by .data[0] ?
This is when I remove sync everywhere .item() is before .data[0]
Cycle:  0
total time 16.41 loading time 10.95, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.78, average cost time with .data[0] 0.02, average cost time with .item() 4.19
cost with data[0] is 3.33067 cost with .item() is 3.33067

Cycle:  1
total time 16.56 loading time 11.09, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.79, average cost time with .data[0] 0.01, average cost time with .item() 4.19
cost with data[0] is 2.20410 cost with .item() is 2.20410

Cycle:  2
total time 16.61 loading time 11.12, cuda transfer time 0.11, forward time: 0.37, cost time 0.02, step time 0.80, average cost time with .data[0] 0.01, average cost time with .item() 4.18
cost with data[0] is 2.05243 cost with .item() is 2.05243

And when .data[0] is before .item()
Cycle:  0
total time 16.26 loading time 10.79, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.78, average cost time with .data[0] 0.01, average cost time with .item() 4.18
cost with data[0] is 3.50083 cost with .item() is 3.50083

Cycle:  1
total time 16.31 loading time 10.85, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.78, average cost time with .data[0] 0.01, average cost time with .item() 4.18
cost with data[0] is 2.17939 cost with .item() is 2.17939

Cycle:  2
total time 16.19 loading time 10.73, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.78, average cost time with .data[0] 0.01, average cost time with .item() 4.19
cost with data[0] is 2.01196 cost with .item() is 2.01196

When I comment out .item() (i,e only .data[0])
Cycle:  0
total time 11.96 loading time 10.69, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.78, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 3.57116 cost with .item() is 0.00000

Cycle:  1
total time 11.92 loading time 10.65, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.78, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 2.27203 cost with .item() is 0.00000

Cycle:  2
total time 11.95 loading time 10.68, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.78, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 2.08692 cost with .item() is 0.00000

Cycle:  3
total time 11.88 loading time 10.61, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.77, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 1.92515 cost with .item() is 0.00000

When I comment out .item() (i,e only .data[0]) and put torch.cuda.synchronize() at the end of the cycle
Cycle 0
total time 16.23 loading time 10.77, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.78, average cost time with .data[0] 0.01, average cost time with .item() 4.18
cost with data[0] is 3.46457 cost with .item() is 0.00000

Cycle:  1
total time 16.06 loading time 10.62, cuda transfer time 0.11, forward time: 0.35, cost time 0.02, step time 0.77, average cost time with .data[0] 0.01, average cost time with .item() 4.20
cost with data[0] is 2.20793 cost with .item() is 0.00000

When I comment out .item() (i,e only .data[0]) and put torch.cuda.synchronize() BEFORE accumulation of cost with .data[0]
Cycle:  0
total time 16.53 loading time 11.06, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 4.97, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 3.53346 cost with .item() is 0.00000

Cycle:  1
total time 16.47 loading time 11.00, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 4.96, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 2.23358 cost with .item() is 0.00000

Cycle:  2
total time 16.43 loading time 10.97, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 4.95, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 2.03888 cost with .item() is 0.00000

Cycle:  3
total time 16.42 loading time 10.95, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 4.97, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 1.92610 cost with .item() is 0.00000

When I comment out .item() (i,e only .data[0]) and put torch.cuda.synchronize() BEFORE accumulation of cost with .data[0] and BEFORE the update phase (i.e step time)
Cycle:  0
total time 16.11 loading time 10.69, cuda transfer time 0.11, forward time: 0.35, cost time 1.05, step time 3.90, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 3.52324 cost with .item() is 0.00000

Cycle:  1
total time 16.10 loading time 10.67, cuda transfer time 0.11, forward time: 0.36, cost time 1.05, step time 3.90, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 2.19925 cost with .item() is 0.00000

Cycle:  2
total time 16.13 loading time 10.70, cuda transfer time 0.11, forward time: 0.36, cost time 1.05, step time 3.90, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 1.97843 cost with .item() is 0.00000

Cycle:  3
total time 16.10 loading time 10.68, cuda transfer time 0.11, forward time: 0.36, cost time 1.04, step time 3.90, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 1.86069 cost with .item() is 0.00000

Finally when I remove all the syncs inside the loop that iterate over the database but put it just before the timer that is activated once the loop is finished:

Cycle:  0
total time 11.96 loading time 10.67, cuda transfer time 0.11, forward time: 0.35, cost time 0.02, step time 0.77, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 3.41078 cost with .item() is 0.00000

Cycle:  1
total time 12.03 loading time 10.73, cuda transfer time 0.11, forward time: 0.35, cost time 0.02, step time 0.77, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 2.16014 cost with .item() is 0.00000

Cycle:  2
total time 12.03 loading time 10.73, cuda transfer time 0.11, forward time: 0.36, cost time 0.02, step time 0.77, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 1.98470 cost with .item() is 0.00000

Cycle:  3
total time 12.00 loading time 10.71, cuda transfer time 0.11, forward time: 0.35, cost time 0.02, step time 0.77, average cost time with .data[0] 0.01, average cost time with .item() 0.00
cost with data[0] is 1.84018 cost with .item() is 0.00000


p.s: In the light of these findings I changed the title of the topic to be less misleading