You are synchronizing before some timers, but skip it inside the inference operation.
What could happen is:
torch.cuda.synchronize() # valid sync, CPU waits until GPU is ready
t0 = 1000 * time.time() # start timer
out = net.inference(input, ratio, patchsize) # execute inference
    # inside inference
    ...... # execute model forward pass
    t1 = 1000*time.time() # start new timer without waiting for the GPU to finish the work
    idx = torch.nonzero(mask.squeeze(1)) # nonzero synchronizes, CPU waits
    t2 = 1000*time.time() # stop timer, which shows the GPU forward time + nonzero op