Very different results on inference between mps and cpu for same input
### System Info

transformers version: 4.22.2
Platform: macOS-12.5-arm64-arm-64bit 
Python version: 3.9.13
Huggingface_hub version: 0.10.0
PyTorch version (MPS?): 1.13.0.dev20221005 (True)


### Who can help?

@patil-suraj, @patrickvonplaten, @LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am running the following code: 

```python
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

model = GPT2LMHeadModel.from_pretrained('gpt2').to(torch.device('mps'))
inputs = tokenizer('the man who is tall',
                   return_tensors='pt').to(torch.device('mps'))

print(inputs)
outputs = model(**inputs).logits
print(outputs[0,0,:])

print()
print()

model = GPT2LMHeadModel.from_pretrained('gpt2').to(torch.device('cpu'))
inputs = tokenizer('the man who is tall',
                   return_tensors='pt').to(torch.device('cpu'))

print(inputs)
outputs = model(**inputs).logits
print(outputs[0,0,:])
```

What I see as output is:

```
{'input_ids': tensor([[1169,  582,  508,  318, 7331]], device='mps:0'), 'attention_mask': tensor([[1, 1, 1, 1, 1]], device='mps:0')}
/opt/homebrew/Caskroom/miniforge/base/envs/mapi/lib/python3.9/site-packages/torch/_tensor_str.py:115: UserWarning: The operator 'aten::nonzero' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
  nonzero_finite_vals = torch.masked_select(
tensor([-14.9901, -14.6213, -17.5936,  ..., -21.4744, -21.1240, -14.7532],
       device='mps:0', grad_fn=<SliceBackward0>)


{'input_ids': tensor([[1169,  582,  508,  318, 7331]]), 'attention_mask': tensor([[1, 1, 1, 1, 1]])}
tensor([-33.1021, -31.8638, -35.0600,  ..., -38.2193, -38.8318, -32.7428],
       grad_fn=<SliceBackward0>)
```

### Expected behavior

I expect the output to be the same (or at least much closer). Am I missing something obvious here? 
Interesting! @pcuenca could you take a look here? <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.