I cannot reproduce the issue.
My Dockerfile:
FROM pytorch/pytorch:1.5-cuda10.1-cudnn7-runtime
RUN pip install request
ENTRYPOINT bash

Building the container via:
docker build -t tmp_build .

Running via:
nvidia-docker run -it --ipc=host tmp_build

Inside the container:
root@...:/workspace# python -c "import torch; print(torch.randn(1, device='cuda'))"
tensor([0.4954], device='cuda:0')

Could you check, what the difference to your workflow might be?