So the reason why our docker image may be a bit bigger is because it contains the necessary packages in order to run with GPU support.
So that something like
docker run --rm -it --gpus all pytorch/pytorch

Will function as expected when you have a GPU that is compatible.