The timing seems to be strange.
I get approx. 31ms and 55ms for the forward and backward pass on a TITAN V, respectively:
device = 'cuda'
model = Unet(in_channels=1, out_channels=1)
model.to(device)
x = torch.randn(1, 1, 540, 540, device=device)

torch.cuda.synchronize()
t0 = time.time()
output = model(x)
torch.cuda.synchronize()
t1 = time.time()
print('fwd {}s'.format(t1 - t0))

loss = output.mean()

torch.cuda.synchronize()
t0 = time.time()
loss.backward()
torch.cuda.synchronize()
t1 = time.time()
print('bwd {}s'.format(t1 - t0))

(1.25s and 1.77s for CPU)
Could your bottleneck be the data loading?
E.g. if you don’t use multiple workers and have a lot of data preprocessing, your GPU might just be starving.