Transformer trainer training crashed with GLM models
### System Info

### Environment

```
- `transformers` version: 4.29.2
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: A100
- Using distributed or parallel set-up in script?: Single GPU
- ```

I suspect it's because the model has no device info attached to it, so when transformer trainer tries to fetch per_device_batch * device but somehow the device is 0 and caused the issue.

See detailed stack trace below:

### Stack trace:

```
/usr/local/lib/python3.10/dist-packages/transformers/trainer.py:1664 in train                    │
│                                                                                                  │
│   1661 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1662 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1663 │   │   )                                                                                 │
│ ❱ 1664 │   │   return inner_training_loop(                                                       │
│   1665 │   │   │   args=args,                                                                    │
│   1666 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1667 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:1909 in _inner_training_loop     │
│                                                                                                  │
│   1906 │   │   │   │   rng_to_sync = True                                                        │
│   1907 │   │   │                                                                                 │
│   1908 │   │   │   step = -1                                                                     │
│ ❱ 1909 │   │   │   for step, inputs in enumerate(epoch_iterator):                                │
│   1910 │   │   │   │   total_batched_samples += 1                                                │
│   1911 │   │   │   │   if rng_to_sync:                                                           │
│   1912 │   │   │   │   │   self._load_rng_state(resume_from_checkpoint)                          │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:633 in __next__           │
│                                                                                                  │
│    630 │   │   │   if self._sampler_iter is None:                                                │
│    631 │   │   │   │   # TODO(https://github.com/pytorch/pytorch/issues/76750)                   │
│    632 │   │   │   │   self._reset()  # type: ignore[call-arg]                                   │
│ ❱  633 │   │   │   data = self._next_data()                                                      │
│    634 │   │   │   self._num_yielded += 1                                                        │
│    635 │   │   │   if self._dataset_kind == _DatasetKind.Iterable and \                          │
│    636 │   │   │   │   │   self._IterableDataset_len_called is not None and \                    │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:677 in _next_data         │
│                                                                                                  │
│    674 │                                                                                         │
│    675 │   def _next_data(self):                                                                 │
│    676 │   │   index = self._next_index()  # may raise StopIteration                             │
│ ❱  677 │   │   data = self._dataset_fetcher.fetch(index)  # may raise StopIteration              │
│    678 │   │   if self._pin_memory:                                                              │
│    679 │   │   │   data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)            │
│    680 │   │   return data                                                                       │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/torch/utils/data/_utils/fetch.py:49 in fetch             │
│                                                                                                  │
│   46 │   def fetch(self, possibly_batched_index):                                                │
│   47 │   │   if self.auto_collation:                                                             │
│   48 │   │   │   if hasattr(self.dataset, "__getitems__") and self.dataset.__getitems__:         │
│ ❱ 49 │   │   │   │   data = self.dataset.__getitems__(possibly_batched_index)                    │
│   50 │   │   │   else:                                                                           │
│   51 │   │   │   │   data = [self.dataset[idx] for idx in possibly_batched_index]                │
│   52 │   │   else:                                                                               │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py:2782 in __getitems__           │
│                                                                                                  │
│   2779 │                                                                                         │
│   2780 │   def __getitems__(self, keys: List) -> List:                                           │
│   2781 │   │   """Can be used to get a batch using a list of integers indices."""                │
│ ❱ 2782 │   │   batch = self.__getitem__(keys)                                                    │
│   2783 │   │   n_examples = len(batch[next(iter(batch))])                                        │
│   2784 │   │   return [{col: array[i] for col, array in batch.items()} for i in range(n_example  │
│   2785                                                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py:2778 in __getitem__            │
│                                                                                                  │
│   2775 │                                                                                         │
│   2776 │   def __getitem__(self, key):  # noqa: F811                                             │
│   2777 │   │   """Can be used to index columns (by string names) or rows (by integer index or i  │
│ ❱ 2778 │   │   return self._getitem(key)                                                         │
│   2779 │                                                                                         │
│   2780 │   def __getitems__(self, keys: List) -> List:                                           │
│   2781 │   │   """Can be used to get a batch using a list of integers indices."""                │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py:2762 in _getitem               │
│                                                                                                  │
│   2759 │   │   format_kwargs = kwargs["format_kwargs"] if "format_kwargs" in kwargs else self._  │
│   2760 │   │   format_kwargs = format_kwargs if format_kwargs is not None else {}                │
│   2761 │   │   formatter = get_formatter(format_type, features=self._info.features, **format_kw  │
│ ❱ 2762 │   │   pa_subtable = query_table(self._data, key, indices=self._indices if self._indice  │
│   2763 │   │   formatted_output = format_table(                                                  │
│   2764 │   │   │   pa_subtable, key, formatter=formatter, format_columns=format_columns, output  │
│   2765 │   │   )                                                                                 │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py:578 in query_table     │
│                                                                                                  │
│   575 │   │   _check_valid_column_key(key, table.column_names)                                   │
│   576 │   else:                                                                                  │
│   577 │   │   size = indices.num_rows if indices is not None else table.num_rows                 │
│ ❱ 578 │   │   _check_valid_index_key(key, size)                                                  │
│   579 │   # Query the main table                                                                 │
│   580 │   if indices is None:                                                                    │
│   581 │   │   pa_subtable = _query_table(table, key)                                             │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py:531 in                 │
│ _check_valid_index_key                                                                           │
│                                                                                                  │
│   528 │   │   │   _check_valid_index_key(min(key), size=size)                                    │
│   529 │   elif isinstance(key, Iterable):                                                        │
│   530 │   │   if len(key) > 0:                                                                   │
│ ❱ 531 │   │   │   _check_valid_index_key(int(max(key)), size=size)                               │
│   532 │   │   │   _check_valid_index_key(int(min(key)), size=size)                               │
│   533 │   else:                                                                                  │
│   534 │   │   _raise_bad_key_type(key)                                                           │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py:521 in                 │
│ _check_valid_index_key                                                                           │
│                                                                                                  │
│   518 def _check_valid_index_key(key: Union[int, slice, range, Iterable], size: int) -> None:    │
│   519 │   if isinstance(key, int):                                                               │
│   520 │   │   if (key < 0 and key + size < 0) or (key >= size):                                  │
│ ❱ 521 │   │   │   raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")       │
│   522 │   │   return                                                                             │
│   523 │   elif isinstance(key, slice):                                                           │
│   524 │   │   pass                                                                               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
IndexError: Invalid key: 4 is out of bounds for size 0
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

### Code to reproduce:

```
!pip install -q bitsandbytes datasets accelerate loralib
!pip install sentencepiece
!pip install -q transformers peft

import torch
import torch.nn as nn
import bitsandbytes as bnb
import datasets
import accelerate
import loralib
import sentencepiece as spm
import transformers

from peft import LoraConfig, get_peft_model


from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-10b-chinese", trust_remote_code=True)
model = AutoModelForSeq2SeqLM.from_pretrained("THUDM/glm-10b-chinese", trust_remote_code=True)
model = model.half().cuda()

ds = ["hello world", "what the heck", "you are not alone", "not toyda", "more or less"]
ds = {"text": ds}
ds = datasets.Dataset.from_dict(ds)
ds = ds.map(lambda x: tokenizer(x["text"], padding=True), batched=True)


config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["query_key_value"],
    lora_dropout=0.05,
    bias="none",
    task_type="CASUAL_LM"
)

model = get_peft_model(model, config)


trainer = transformers.Trainer(
    model=model,
    train_dataset=ds,
    args=transformers.TrainingArguments(
        per_device_train_batch_size=1,
        gradient_accumulation_steps=1,
        warmup_steps=0,
        num_train_epochs=2,
        learning_rate=3e-4,
        fp16=True,
        logging_steps=1,
        output_dir='outputs',
        save_total_limit=2,
    ),
    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False)
)
model.config.use_cache = False  # silence the warnings. Please re-enable for inference!
trainer.train()

```

### Expected behavior

Expect not to crash
The problem lies within your dataset, nothing to do with the `Trainer` :-)<|||||>Hi @sgugger, I'm new to `Transformer.Trainer`, wonder what's the issue here? how should I setup the dataset? Thanks!

I thought the tokenizer should tokenize the text and return a dict with `input_ids` in it. then `transformers.DataCollatorForLanguageModeling` should map `input_ids` to `labels` correctly?

Wonder if there is any example about using custom dataset?<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.