Thanks for the suggestion. I’ve changed the profiling code to the following:
### Profiling model run time

torch.cuda.synchronize()
start = time.perf_counter()

res = model(torch.from_numpy(img).type(torch.cuda.FloatTensor) / 255)

torch.cuda.synchronize()
end = time.perf_counter()
print(f'Run time: {end - start}')

### Profiling GPU-CPU copy time
torch.cuda.synchronize()
t1 = time.perf_counter()

pred = res['out'].cpu().detach().numpy()[0][0]
                    
torch.cuda.synchronize()
t2 = time.perf_counter()
print(f'CPU copy time: {t2 - t1}')

The result is (inside docker):
Run time: 10.947372228954919
CPU copy time: 0.0007527889683842659
I did another profiling with the following code to see how long the synchronization takes:
torch.cuda.synchronize()
start = time.perf_counter()

res = model(torch.from_numpy(img).type(torch.cuda.FloatTensor) / 255)

mid = time.perf_counter()

torch.cuda.synchronize()
end = time.perf_counter()
print(f'Run time: {end - start}')
print(f'Second sync call time: {end - mid}')

And got:
Second sync call time: 10.835095343994908
From my perspective the 2nd torch.cuda.synchronize() is taking a long time in the docker. Do you have any suggestions? Thanks.