I retimed everything myself using torch.cuda.synchronize() (see the following code and output) and my forward pass would be taking ~0.3s whereas my backward pass is taking ~3.9s. Are these the correct timings and if so is there anything I can do ?
    for i, data in enumerate(dataloader):
        t0 = time()
        data_gpu = data.cuda()
        t1 = time()
        total_time += t1 - t0
        print("Loading data on GPU", t1 - t0)

        output = model(data_gpu)
        torch.cuda.synchronize()
        t2 = time()
        print("Real time forward pass", t2 - t1)
        labels = torch.Tensor([0] * batch_size).long().cuda()
        loss = criterion(output, labels)
        torch.cuda.synchronize()
        t3 = time()
        print("Real time for loss computation", t3 - t2)
        loss.backward()
        torch.cuda.synchronize()
        t4 = time()
        print("Real time for backward pass", t4 - t3)

outputs:
Real time forward pass 0.33179545402526855
Real time for loss computation 0.0008327960968017578
Real time for backward pass 3.920534610748291
Loading data on GPU 0.008051156997680664
Real time forward pass 0.26632165908813477
Real time for loss computation 0.0001857280731201172
Real time for backward pass 3.930469512939453
Loading data on GPU 0.00805807113647461
Real time forward pass 0.2658727169036865
Real time for loss computation 0.00015234947204589844
Real time for backward pass 3.920789957046509
Loading data on GPU 0.008044242858886719
Real time forward pass 0.2670416831970215
Real time for loss computation 0.00017786026000976562
Real time for backward pass 3.932731866836548
Loading data on GPU 0.008074760437011719
Real time forward pass 0.2663881778717041
Real time for loss computation 0.00015616416931152344
Real time for backward pass 3.9343338012695312
Loading data on GPU 0.008038997650146484
Real time forward pass 0.2661571502685547
Real time for loss computation 0.00015997886657714844
Real time for backward pass 3.925018072128296