Hi,
In you example, you could replace the transpose function by any function in torch, you would get the same behavior.
The transpose operation does not actually touches the tensor data and just work on the metadata. The code to do that on cpu and gpu is exactly the same and never touches the gpu. The runtimes that you see in your test is just the overhead of the python loop + calling into c code (in your case the c code does almost nothing).
The gpu version is slightly slower because the cuda library has to get its state before calling the functions which slows it slightly compared to the pure cpu version.
This code sample is slow only because of the python loop which calls c functions. To make it faster, you need to find a way to remove this loop. If in your case, you want to transpose a bunch of matrices, you could for example stack them in a single tensor and then call transpose on this tensor.