Anything but plain "greedy" search "not implemented for 'Half'"
### System Info

`transformers-cli env`=
```
Traceback (most recent call last):
  File "/home/user/.local/bin/transformers-cli", line 5, in <module>
    from transformers.commands.transformers_cli import main
  File "/home/user/.local/lib/python3.10/site-packages/transformers/commands/transformers_cli.py", line 24, in <module>
    from .pt_to_tf import PTtoTFCommand
  File "/home/user/.local/lib/python3.10/site-packages/transformers/commands/pt_to_tf.py", line 21, in <module>
    from datasets import load_dataset
ModuleNotFoundError: No module named 'datasets'
```
https://colab.research.google.com/drive/1YORPWx4okIHXnjW7MSAidXN29mPVNT7F?usp=sharing

### Who can help?

@sgugger, @patil-suraj

Im following:
https://colab.research.google.com/drive/1YORPWx4okIHXnjW7MSAidXN29mPVNT7F
from:
https://huggingface.co/blog/hf-bitsandbytes-integration

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I love being able to use the int8 models! And being able to run the big LLMs on Colab. However, I just discovered that only the default search algo works. This is too bad, because for a chat application, the TopK sampling provides much more natural variation.

Beam:
RuntimeError: "log_softmax_lastdim_kernel_impl" not implemented for 'Half'
Topk:
RuntimeError: "topk_cpu" not implemented for 'Half'

### Expected behavior

being able to use TopK and TopP sampling on int8 optimized models
Hey @auwsom ! 
Thanks for your message! Indeed it would be better to support all the possible sampling procedures for 8-bit models 💪 
There is definitely something around half-precision logits and `generate` that needs a closer look! 
Also this seems to be a duplicate of https://github.com/TimDettmers/bitsandbytes/issues/42#issuecomment-1272877078 - so tagging the issue here 

Will look into it ASAP! <|||||>Hey @auwsom !
Thanks for your patience!
It appears that the workaroud is pretty much straightforward, could you run `generate` with `inputs_ids` set to a GPU device? For example by making sure that: 
```
input_ids = input_ids.to('cuda')
```
`generate` yields an error since by instantiating a model with `device_map=auto` forces the output of the model to be on the same device as the input. In the snippet in https://github.com/TimDettmers/bitsandbytes/issues/42#issuecomment-1272877078 the `input_ids` are set on the `cpu`. I believe that making sure that these are on the GPU should do the trick for you, before waiting a proper fix to be merged in #19468 ? Could you confirm that this workaround fixes your issue? Thanks!<|||||>@younesbelkada yes, this works on the example notebook on Colab. Thanks!