Pretrained model outputs all zeros on GPU in a docker environment
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- `transformers` version: 4.12.3
- Platform: Linux-4.15.0-161-generic-x86_64-with-glibc2.27
- Python version: 3.8.0
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help
@patil-suraj

## Information

Model I am using (Bert, XLNet ...): GPT-Neo

The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is: N/A
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
from transformers import GPTNeoForCausalLM, GPT2Tokenizer
import torch

model = GPTNeoForCausalLM.from_pretrained("EleutherAI/gpt-neo-125M")
tokenizer = GPT2Tokenizer.from_pretrained("EleutherAI/gpt-neo-125M")

prompt = 'I love Pixar.'
inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs['input_ids']
attention_mask = inputs['attention_mask']

a1 = model(input_ids=input_ids, attention_mask=attention_mask, labels=input_ids)
print('a1', a1.logits) # CPU output

input_ids_cuda = input_ids.cuda()
attention_mask_cuda = attention_mask.cuda()
model.to('cuda:0')
a2 = model(input_ids=input_ids_cuda, attention_mask=attention_mask_cuda, labels=input_ids_cuda)
print('a2', a2.logits) # GPU output: All zeros
```

Output:
```
a1 tensor([[[ -7.9161,  -5.5552,  -7.7819,  ..., -16.8574, -11.3656,  -7.8918],
         [ -5.3131,  -5.2017,  -8.3746,  ..., -15.0816, -10.2103,  -7.4852],
         [  0.9403,  -5.3730,  -9.5152,  ..., -18.6976, -16.9710,  -7.5517],
         [ -6.6690,  -6.1381,  -8.9958,  ..., -24.4115, -13.6879,  -3.7547]]],
       grad_fn=<UnsafeViewBackward>)
a2 tensor([[[0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.]]], device='cuda:0',
       grad_fn=<UnsafeViewBackward>)
```

## Expected behavior
GPU outputs should be the same as or similar to CPU outputs.
I tried to use a local conda environment and there was no issue. It seems like there is something to do with the docker environment. Here is the docker file that I used:
```
ARG CUDA_DOCKER_VERSION=11.2.2-devel-ubuntu18.04
FROM nvidia/cuda:${CUDA_DOCKER_VERSION}

# Arguments for the build. CUDA_DOCKER_VERSION needs to be repeated because
# the first usage only applies to the FROM tag.
# TensorFlow version is tightly coupled to CUDA and cuDNN so it should be selected carefully
ARG CUDA_DOCKER_VERSION=11.2.2-devel-ubuntu18.04
ARG PYTORCH_VERSION=1.8.1+cu111
ARG PYTORCH_LIGHTNING_VERSION=1.2.9
ARG TORCHVISION_VERSION=0.9.1+cu111
ARG CUDNN_VERSION=8.1.1.33-1+cuda11.2
ARG NCCL_VERSION=2.8.4-1+cuda11.2
ARG PYTHON_VERSION=3.8

# Set default shell to /bin/bash
SHELL ["/bin/bash", "-cu"]

RUN bash -c "apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub"

RUN apt update

RUN apt-get update && apt-get install -y --allow-downgrades --allow-change-held-packages --no-install-recommends \
        build-essential \
        cmake \
        g++-7 \
        git \
        curl \
        vim \
        wget \
        ca-certificates \
        libcudnn8=${CUDNN_VERSION} \
        libnccl2=2.8.4-1+cuda11.2 \
        libnccl-dev=2.8.4-1+cuda11.2 \
        libjpeg-dev \
        libpng-dev \
        python${PYTHON_VERSION} \
        python${PYTHON_VERSION}-dev \
        python${PYTHON_VERSION}-distutils \
        librdmacm1 \
        libibverbs1 \
        ibverbs-providers \
        openjdk-8-jdk-headless \
        openssh-client \
        openssh-server \
    && apt-get clean && rm -rf /var/lib/apt/lists/*

# Install Open MPI
RUN wget --progress=dot:mega -O /tmp/openmpi-3.0.0-bin.tar.gz https://github.com/horovod/horovod/files/1596799/openmpi-3.0.0-bin.tar.gz && \
    cd /usr/local && \
    tar -zxf /tmp/openmpi-3.0.0-bin.tar.gz && \
    ldconfig && \
    mpirun --version
    
# Allow OpenSSH to talk to containers without asking for confirmation
RUN mkdir -p /var/run/sshd
RUN cat /etc/ssh/ssh_config | grep -v StrictHostKeyChecking > /etc/ssh/ssh_config.new && \
    echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config.new && \
    mv /etc/ssh/ssh_config.new /etc/ssh/ssh_config

RUN ln -s /usr/bin/python${PYTHON_VERSION} /usr/bin/python

RUN curl -O https://bootstrap.pypa.io/get-pip.py && \
    python get-pip.py && \
    rm get-pip.py

# Install PyTorch
RUN pip install --no-cache-dir \
    torch==${PYTORCH_VERSION} \
    torchvision==${TORCHVISION_VERSION} \
    -f https://download.pytorch.org/whl/${PYTORCH_VERSION/*+/}/torch_stable.html
RUN pip install --no-cache-dir pytorch_lightning==${PYTORCH_LIGHTNING_VERSION}

RUN pip install --no-cache-dir future typing packaging

RUN ldconfig /usr/local/cuda/targets/x86_64-linux/lib/stubs && \
      bash -c "HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITH_PYTORCH=1 pip install --no-cache-dir -v horovod==v0.22.1" && \
      horovodrun --check-build && \
      ldconfig

RUN dpkg --add-architecture i386
RUN apt-get update
RUN apt install -y libsm6 libxext6 libxrender-dev libfontconfig1 libglib2.0-0

# Check all frameworks are working correctly. Use CUDA stubs to ensure CUDA libs can be found correctly
# when running on CPU machine
# RUN ldd /home/ubuntu/horovod/local/lib/python2.7/site-packages/horovod/common/mpi_lib.so
RUN ldconfig /usr/local/cuda/targets/x86_64-linux/lib/stubs && \
    python -c "import horovod.torch as hvd; hvd.init()" && \
    ldconfig

# Python packages
RUN pip install \
    numpy \
    tqdm \
    pandas \
    scikit-learn \
    matplotlib

RUN pip install ipython \
    jupyterlab \
    ipywidgets

RUN pip install transformers

RUN pip install datasets
```<|||||>hi! What's your local PyTorch and transformers version?<|||||>I tried this with pytorch `1.8.1` but couldn't re-produce <|||||>My local pytorch and transformer versions are the same as the docker one.
Strangely in docker this issue happens but not in a conda environment.

On Mon, Nov 8, 2021 at 09:27 Suraj Patil ***@***.***> wrote:

> I tried this with pytorch 1.8.1 but couldn't re-produce
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/14308#issuecomment-963214174>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAIVUFJX5PN5W2U5BEZWZMTUK7M57ANCNFSM5HQMBGUQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
-- 
Sent from my iPhone
<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.