I’m having an issue of slow .to(device) transfer of a single batch. If I understood correctly, dataloader should be sampled from in the main training loop and only then (when the whole batch is gathered) should be transferred to gpu with .to(device) method of the batch tensor?
My batch size is 32 samples x 64 features x 1000 length x 4 bytes (float32) / (1024*1024) = ~7MB and I’m using 1080Ti graphics card. 7MB tensor.to(device) should be pretty much instantaneous? Or am I missing something in my calculation?
bottleneck profiler data is available here https://pastebin.com/dvQe6Y2Q 18 (copy raw data to gedit/notepad for easy view)
here is the code that was profiled:
 it = iter(dataloader)
    for i in range(5):
        time_s = time.time()
        sample = next(it)
        optimizer.zero_grad()
        data = sample[0].to(device)
        iz = model(data)
        ctc_loss = nn.CTCLoss(zero_infinity=True)
        loss = ctc_loss(iz.transpose(0, 1).transpose(0, 2), sample[1], sample[2], sample[3])
        loss.backward()
        optimizer.step()
        time_e = time.time()
        print("Time ", time_e - time_s)