Hello, I’m trying to make my model (deeplabv3-resnet101) do inference inside a Docker container. I tested the inference code both inside/outside the container but got a significant time difference at model.cpu() call.
Here’s the basic code snippet:
model=torch.load('path_to_model.pt')
model.eval()
for image in images:
    image = image.reshape(1, 3, 640, 640)
    res = model(torch.from_numpy(image).type(torch.cuda.FloatTensor) / 255.0)
    pred = res['out'].cpu().detach().numpy()[0][0]
    # do other stuff...


Docker base: pytorch/pytorch:1.6.0-cuda10.1-cudnn7-runtime

GPU: GeForce RTX 2080Ti
I made sure model inference happens on cuda:0. So the inference time inside/outside docker is almost the same
Calling res['out'].cpu() (a [1,1,640,640] tensor) takes 0.08s outside container and 10s inside container. I tried adding torch.cuda.sychronize() as in here but didn’t make a difference.

So what could be the reason causing this difference and how do I solve it? Thanks!