when I write some model about CNN, I found the transpose op is too slow in GPU, even slower than CPU
here is some my test code
test1:
import torch
import time
from torch.autograd import Variable

x = Variable(torch.randn(100,500))
cputimes = []
for sampl in (1000, 10000, 100000, 1000000):
    start = time.time()
    for i in range(sampl):
        y = torch.transpose(x,0,1)
    end = time.time()
    cputimes.append(end-start)
print(cputimes)

x = x.cuda(device_id=2)
gputimes = []
for sampl in (1000, 10000, 100000, 1000000):
    start = time.time()
    for i in range(sampl):
        y = torch.transpose(x,0,1)
    end = time.time()
    gputimes.append(end-start)
print(gputimes)

output:
[0.00479888916015625, 0.047800540924072266, 0.5636386871337891, 4.8213441371917725]
[0.0057294368743896484, 0.056331634521484375, 0.5558302402496338, 5.78531289100647]
test2:
In [16]: torch.cuda.set_device(2)

In [17]: %timeit torch.transpose(torch.FloatTensor(20,100),1,0)
The slowest run took 26.26 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 1.72 µs per loop

In [18]: %timeit torch.transpose(torch.cuda.FloatTensor(20,100),1,0)
The slowest run took 21.21 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 3.05 µs per loop