Actually, I did some more analysis of the timing. It turns out, most of the time spent in the routine is in the loss.backward() operation, not when computing the forward nor when loading the data with .cuda().  Here is a snippet of the code inside my training loop.
for data in dataloader:
    data = data.cuda()
    data = torch.torch.autograd(data, requires_grad=True)
    output = model(data)
    loss  = criterion(output, labels)
    predictions = (output.detach() > 0).int()
    accuracy = (labels.detach().int() == predictions).sum().item()/batch_size
    optimizer.zero_grad()
    loss.backward() ### backward takes 100x longer than forward in torch 1.4.0
    optimizer.step()