OneFormer processor does not return correctly formatted class_labels tensors
### System Info

- `transformers` version: 4.29.0.dev0
- Platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): 2.11.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.5.3 (cpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@amyeroberts

I'm trying to finetune `shi-labs/oneformer_ade20k_swin_tiny` on my own dataset. I've hit two problems, one with the docs and one with the actual library code for image processing.

1. There are no docs on fine-tuning or training the OneFormer model on this page: https://huggingface.co/docs/transformers/model_doc/oneformer . So I relied on investigating this test
https://github.com/huggingface/transformers/blob/main/tests/models/oneformer/test_modeling_oneformer.py#L364
2. The train test doesn't actually use the OneFormer Processor that is used in all of the inference examples in  https://huggingface.co/docs/transformers/model_doc/oneformer

I think this is because the OneFormer Processor and the train test produce differently formatted class labels. In the train test, the class_labels are created from scratch here: https://github.com/huggingface/transformers/blob/main/tests/models/oneformer/test_modeling_oneformer.py#L106

When training, it's expected that class_labels is a Tensor shaped like [batch_size, num_classes], where a particular element in a batch would have [1,0,0,0] to represent the 0th class.

But the OneFormer processor returns a list of tensors with values greater than 1:  [tensor([0, 3])]

This eventually leads to an error here https://github.com/huggingface/transformers/blob/v4.28.1/src/transformers/models/oneformer/modeling_oneformer.py#L306 where we get index out of bounds but it's a CUDA assert error.

I think this should be rectified by including a training example in the docs and changing the test and OneFormer Processor so that they work when training the model.

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. adapt test_training to create class_labels with OneFormerProcessor instead of from scratch
2. run the adapted test_training

I have a failing example at https://github.com/developmentseed/slickformer but it takes quite a bit of setup and the datasets in't public yet.

### Expected behavior

I'd expect OneFormer Processor to return a class_labels entry in the dict that has the same expected output as `class_labels` in `test_training`. To summarize, the class_labels need to be one hot encoded for training, but OneFormerProcessor isn't doing this.
Hi,

I'd recommend taking a look at the MaskFormer/Mask2Former notebooks regarding fine-tuning on custom data: https://github.com/NielsRogge/Transformers-Tutorials/tree/master/MaskFormer. As the API of OneFormer is identical, except that it has one additional `task_inputs` input which you need to prepare as well.<|||||>Hi @NielsRogge @amyeroberts 
Actually I  was following your MaskFormer and Mask2Former tutorials 
and my task was to finetune on Semantic information and have an instance level prediction. Where Mask2Former was performing well.
Still Problems I have mentioned on Closed Issue : https://github.com/huggingface/transformers/issues/21644
My main request was Can you also Write Training Tutorials for OneFormer in MaskFormer, Text input part is creating the problem and the segmentation_maps parameter. 

```
    779 annotation_classes = label["classes"]
    780 annotation_masks = label["masks"]
--> 782 texts = ["a semantic photo"] * self.num_text
    783 classes = []
    784 masks = []

TypeError: can't multiply sequence by non-int of type 'NoneType'

```
```
preprocessor = OneFormerImageProcessor.from_pretrained(config.MODEL_PATH)
preprocessor.num_text = 2
preprocessor.num_classes = 2
preprocessor.ignore_index=3 
preprocessor.do_reduce_labels=False 
preprocessor.do_resize=False
preprocessor.from_json_filedo_rescale=False
preprocessor.do_normalize=True
preprocessor.image_mean = config.MEAN
preprocessor.image_std = config.STD
```
after introducing num_text:
```

 972     num_class_obj[cls_name] = 0
    974 for i, label in enumerate(annotations):
--> 975     task = task_inputs[i]
    976     if task == "semantic":
    977         classes, masks, texts = self.get_semantic_annotations(label, num_class_obj)

IndexError: list index out of range
```

```
def collate_fn(batch):
    inputs = list(zip(*batch))
    images = inputs[0]
    segmentation_maps = inputs[1]
    # this function pads the inputs to the same size,
    # and creates a pixel mask
    # actually padding isn't required here since we are cropping
   
    batch = preprocessor(
        images,
        task_inputs=["semantic"],
        segmentation_maps=segmentation_maps,

         return_tensors="pt",
    )
    
    return batch
```<|||||>Any solution? I'm having the same or similar issues when changing the MaskFormerImageProcessor to OneFormerProcessor in the tutorial https://github.com/NielsRogge/Transformers-Tutorials/blob/master/MaskFormer/Fine-tuning/Fine_tuning_MaskFormer_on_a_panoptic_dataset.ipynb