I did some tests on a single GPU and the results were the expected ones (e.g., if I reduce the batch size and the number of input of images so that the number of iterations remains the same, then the execution time is smaller).
The execution time in all the tests is collected outside pytorch, so all the copy/synchronization issues should be taken into account.