self_attention_mask
### System Info


- `transformers` version: 4.29.2
- Platform: Linux-5.4.17-2136.318.7.1.el7uek.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.15.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@sg

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I call the `bigcode/starcoderbase` model, the output comes `nan`. I debugged the code and found that it happens in the `modeling_gpt_bigcode.py` where it gets the attention output in `GPTBigCodeAttention._attn()`. 

Moving back to find the root of the issue I find that the `attention-mask` is edited in `GPTBigCodeModel.forward()` function as shown below. I could not understand the reason behind this modification of the `attention_mask` so though better I ask it here to see if it is intended or if there is any bug in that.
 

```
        # Self-attention mask.
        query_length = input_shape[-1]
        key_length = past_length + query_length
        self_attention_mask = self.bias[None, key_length - query_length : key_length, :key_length]

        if attention_mask is not None:
            self_attention_mask = self_attention_mask * attention_mask.view(batch_size, 1, -1).to(
                dtype=torch.bool, device=self_attention_mask.device
            )

        # MQA models: (batch_size, query_length, n_heads, key_length)
        # MHA models: (batch_size, n_heads, query_length, key_length)
        attention_mask = self_attention_mask.unsqueeze(2 if self.multi_query else 1)
```

Note that this modification results in attention mask like `[False, False, ...., False]` which using it in the `_attn()` functions results in:
```
attn_weights = torch.where(attention_mask, attn_weights, mask_value) 
	attn_weights[1,1,1,:].max() = -\infty
attn_weights = softmax(attn_weights, dim=-1)
	attn_weights[1,1,1,:].max() = [nan, ..., nan] (size of 2048)
```



### Expected behavior

As above. 
cc @ArthurZucker @younesbelkada <|||||>Hey @oroojlooy thanks for opening an issue. Sorry didn't really have time to dive into this, could you try with the latest version of `transformers`. Could you also share you script on how you called the model! 🤗 