Let's not cast them all
# What does this PR do?

This PR is an alternative solution (and cleaner) to https://github.com/huggingface/transformers/pull/18467
An issue has been found when running this script:
```
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Salesforce/codegen-2B-mono")
model = AutoModelForCausalLM.from_pretrained("Salesforce/codegen-2B-mono", device_map="auto", torch_dtype=torch.float16)

text = "def quicksort(l):"

encoded_input = tokenizer(text, return_tensors='pt')
output_sequences = model.generate(input_ids=encoded_input['input_ids'], attention_mask=encoded_input['attention_mask'])
print(tokenizer.decode(output_sequences[0], skip_special_tokens=True))
```
Since `torch_dtype=torch.float16` will cast all parameters of the models including the buffers, this will include also causal masks for some models. 

In some niche cases those buffers are in `uint` or `bool` instead of `int`. This PR should address this issue and check if the parameter is either an `uint`, `int` or `bool` before casting it.

cc @sgugger 

Ran codegen slow tests and the tests are passing, let me know if we need more checks! 


Thanks also for giving me the right pointer to the rootcause!<|||||>_The documentation is not available anymore as the PR was closed or merged._