You are most likely just measuring the kernel launch times in your second code snippet.
To properly time a segment, you would have to synchronize before starting and stopping the timer.
E.g. this code should show a high duration in the actual forward pass:
while i < 10: 
    with torch.no_grad():
        image_i = torch.from_numpy(dataset).float().cuda()/255.0
        torch.cuda.synchronize()
        a = time.perf_counter()
        pred = torch.argmax(network(image_i)["seg"][0] , dim=0)
        torch.cuda.synchronize()
        b = time.perf_counter()
        pred.byte().any()>0
        print(b-a)
        
        i=i+1