device: gtx 1070
tensor info:
variable shape of approx: [150, 6]
dtype: torch.int16
I’ve also tested float32 but got the same results
tq = tqdm(trainset)
for i in tq:
    optimizer.zero_grad()
    tic = time.time()
    pred = model(i[0].to(device))
    toc = time.time()
    #This is what is taking so long
    i[1] = i[1].cuda()#.to(device)
    tooc = time.time()
    loss = test(pred, i[1])
    
    loss[0].backward()
    optimizer.step()
    times = loss[1]
    times['inference'] = int((toc-tic) * 1000)
    times['to cuda!?!?'] = int((tooc-toc) * 1000)
    tq.set_postfix(times)

output:
build anchors=40, big loop=114, inference=90, to cuda!?!?=573
units: ms.