Training DetrForObjectDetection failed in a multiple-GPU environment.
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.2
- Platform: Linux-5.4.0-80-generic-x86_64-with-glibc2.27
- Python version: 3.8.0
- PyTorch version (GPU?): 1.9.0+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <yes>
- Using distributed or parallel set-up in script?: <yes>

### Who can help
@NielsRogge


## Information

Model I am using (Bert, XLNet ...): DetrForObjectDetection

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
Steps to reproduce the behavior:

1. Save this script as `run.py` It is the same as https://colab.research.google.com/drive/1oIHGwr1U0sw-6KW-MG60s-ksXA-kYyUO?usp=sharing#scrollTo=VCr7Y7zW5a2a
2. Put sample.json, sample.jpg, sample2.jpg in [detr_samples.tar.gz](https://github.com/huggingface/transformers/files/7019224/detr_samples.tar.gz)  to the same directory.

```python
from typing import Any, Dict, List, Union
from dataclasses import dataclass
import torch
from torchvision.datasets import CocoDetection

from transformers import (
    DetrConfig,
    DetrFeatureExtractor,
    DetrForObjectDetection,
    HfArgumentParser,
    Trainer,
    TrainingArguments,
)

class DetrTrainer(Trainer):
  # Overwrite _prepare_inputs method to make sure dict is also placed on device
  def _prepare_inputs(self, inputs: Dict[str, Union[torch.Tensor, Any]]) -> Dict[str, Union[torch.Tensor, Any]]:
        """
        Prepare :obj:`inputs` before feeding them to the model, converting them to tensors if they are not already and
        handling potential state.
        """
        for k, v in inputs.items():
            if isinstance(v, torch.Tensor):
                kwargs = dict(device=self.args.device)
                if self.deepspeed and inputs[k].dtype != torch.int64:
                    # NLP models inputs are int64 and those get adjusted to the right dtype of the
                    # embedding. Other models such as wav2vec2's inputs are already float and thus
                    # may need special handling to match the dtypes of the model
                    kwargs.update(dict(dtype=self.args.hf_deepspeed_config.dtype()))

                inputs[k] = v.to(**kwargs)
            # labels are a list of dictionaries, each dictionary being a COCO annotation
            if isinstance(v, list):
              for annotation_dict in v:
                  for key, value in annotation_dict.items():
                    annotation_dict[key] = value.to(self.args.device)

        if self.args.past_index >= 0 and self._past is not None:
            inputs["mems"] = self._past

        return inputs


def load_category(category):
    id2label = {}
    label2id = {}
    maxid = 0
    for k, v in category.items():
        id2label[int(k)] = v["name"]
        label2id[v["name"]] = int(k)
        maxid = max(maxid, int(k))
    for i in range(maxid):
        if not (i in id2label):
            id2label[i] = None
    return id2label, label2id


class DetrData(CocoDetection):
    def __init__(self, img_folder, annotations, feature_extractor, train=True):
        super(DetrData, self).__init__(img_folder, annotations)
        self.feature_extractor = feature_extractor

    def __getitem__(self, idx):
      # read in PIL image and target in COCO format
      img, target = super(DetrData, self).__getitem__(idx)

      # preprocess image and target (converting target to DETR format, resizing + normalization of both image and target)
      image_id = self.ids[idx]
      target = {'image_id': image_id, 'annotations': target}
      encoding = self.feature_extractor(images=img, annotations=target, return_tensors="pt")
      encoding["pixel_values"] = encoding["pixel_values"].squeeze() # remove batch dimension
      encoding["labels"] = encoding["labels"][0] # remove batch dimension
      return encoding

@dataclass
class DataCollatorDetr:

    feature_extractor: DetrFeatureExtractor

    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
        pixel_values = [item["pixel_values"] for item in features]
        encoding = self.feature_extractor.pad_and_create_pixel_mask(pixel_values, return_tensors="pt")
        encoding["labels"] = [item["labels"] for item in features]
        return encoding


def main():
    training_args = TrainingArguments(output_dir=".")
    feature_extractor = DetrFeatureExtractor()
    train_dataset = DetrData(img_folder=".", annotations="sample.json", feature_extractor=feature_extractor)
    id2label, label2id = load_category(train_dataset.coco.cats)
    config = DetrConfig.from_pretrained("facebook/detr-resnet-50")
    config.id2label = id2label
    config.label2id = label2id

    model = DetrForObjectDetection.from_pretrained(
        "facebook/detr-resnet-50",
        config=config)

    # Initialize our Trainer
    trainer = DetrTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        tokenizer=feature_extractor,
        data_collator=DataCollatorDetr(feature_extractor=feature_extractor),
    )

    train_result = trainer.train()

if __name__ == "__main__":
    main()
```

3.  Run by `python run.py` in a multiple-GPU environment.
Then `IndexError` is caused.

```
Traceback (most recent call last):
  File "run.py", line 112, in <module>
    main()
  File "run.py", line 109, in main
    train_result = trainer.train()
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/transformers/trainer.py", line 1286, in train
    tr_loss += self.training_step(model, inputs)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/transformers/trainer.py", line 1779, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/transformers/trainer.py", line 1811, in compute_loss
    outputs = model(**inputs)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 168, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 178, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
    raise self.exc_type(msg)
IndexError: Caught IndexError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/transformers/models/detr/modeling_detr.py", line 1430, in forward
    loss_dict = criterion(outputs_loss, labels)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/transformers/models/detr/modeling_detr.py", line 2004, in forward
    indices = self.matcher(outputs_without_aux, targets)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/transformers/models/detr/modeling_detr.py", line 2132, in forward
    indices = [linear_sum_assignment(c[i]) for i, c in enumerate(cost_matrix.split(sizes, -1))]
  File "/home/christopher/detr_samples/env/lib/python3.8/site-packages/transformers/models/detr/modeling_detr.py", line 2132, in <listcomp>
    indices = [linear_sum_assignment(c[i]) for i, c in enumerate(cost_matrix.split(sizes, -1))]
IndexError: index 1 is out of bounds for dimension 0 with size 1
```
It works fine with a single GPU.


## Expected behavior
Successfully complete training.
<!-- A clear and concise description of what you would expect to happen. -->

Hi,

As explained in the [docs](https://huggingface.co/transformers/model_doc/detr.html):
> If you want to train the model in a distributed environment across multiple nodes, then one should update the num_boxes variable in the DetrLoss class of modeling_detr.py. When training on multiple nodes, this should be set to the average number of target boxes across all nodes, as can be seen in the original implementation here.

I had to remove the distributed training-related code from the modeling file, which is perhaps a bit unfortunate, because now people need to fork the library in order for DETR to work properly in a distributed environment. cc @sgugger @LysandreJik <|||||>Hi, thank you for the quick reply.

> When training on multiple nodes, this should be set to the average number of target boxes across all nodes, 

I'd like to ask you two questions.
1. Do I need to insert `num_boxes / (the number of nodes)`  after the following line https://github.com/huggingface/transformers/blob/master/src/transformers/models/detr/modeling_detr.py#L2013 ? 
For example, if I'm training on two GPUs, should I insert `num_boxes = num_boxes / 2` after the line?

2. The error occurs at https://github.com/huggingface/transformers/blob/master/src/transformers/models/detr/modeling_detr.py#L2004 which is before the declaration of `num_boxes`. Could you tell me more about how to solve this error?
<|||||>For now, the code has not been tested to work on multiple GPUs, so this is a good opportunity to make it work. We can perhaps write a guide on which things to take into account. 

> Do I need to insert num_boxes / (the number of nodes) after the following line https://github.com/huggingface/transformers/blob/master/src/transformers/models/detr/modeling_detr.py#L2013 ?
For example, if I'm training on two GPUs, should I insert num_boxes = num_boxes / 2 after the line?

The [original implementation](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/models/detr.py#L230-L232) used the following code:
```
if is_dist_avail_and_initialized():
   torch.distributed.all_reduce(num_boxes)
num_boxes = torch.clamp(num_boxes / get_world_size(), min=1).item()
```
with

```
import torch.distributed as dist

def is_dist_avail_and_initialized():
    if not dist.is_available():
        return False
    if not dist.is_initialized():
        return False
    return True

def get_world_size():
    if not is_dist_avail_and_initialized():
        return 1
    return dist.get_world_size()
```
The world size is 2 if you're training on a single node with 2 GPUs, so you can divide them indeed by 2.

> The error occurs at https://github.com/huggingface/transformers/blob/master/src/transformers/models/detr/modeling_detr.py#L2004 which is before the declaration of num_boxes. Could you tell me more about how to solve this error?

This could have to do with the `targets` not being on the proper devices, which is the responsibility of the `Trainer`. In the original implementation, they use `DistributedSampler`. Can you perhaps print the `sizes` that are computed right before it? These should be a list containing the number of bounding boxes for every example in the batch. <|||||>> For now, the code has not been tested to work on multiple GPUs, so this is a good opportunity to make it work. We can perhaps write a guide on which things to take into account.

It would be great if you could support multiple GPUs.

> The world size is 2 if you're training on a single node with 2 GPUs, so you can divide them indeed by 2.

I found out that if you do it manually, divide by the number of GPUs.

> This could have to do with the targets not being on the proper devices, which is the responsibility of the Trainer. In the original implementation, they use DistributedSampler. Can you perhaps print the sizes that are computed right before it? These should be a list containing the number of bounding boxes for every example in the batch.

The reason for the error is that DistributedSampler does not support `labels` data. 
Thank you very much.<|||||>1. If I want to extend this to panoptic segmentation using coco stuff classes, how should I change the class config to do it.
I have only 1things categories balloon and 53 stuff categories from coco dataset
2. How do I freeze the weights for training mask head for 25 epochs
3. How do we edit the classifier layer of the model say by default this will have 92 class, but from the above example if I have only 2 class (balloon, 'N/A') how should I change them?
<|||||>> If I want to extend this to panoptic segmentation using coco stuff classes, how should I change the class config to do it.
I have only 1things categories balloon and 53 stuff categories from coco dataset

If you want to do panoptic segmentation, you first need to load the model as follows:

```
from transformers import DetrForSegmentation

# specify a custom number of classes
model = DetrForSegmentation.from_pretrained("facebook/detr-resnet-50-panoptic", num_labels=54, ignore_mismatched_sizes=True)
```
You can possibly also add the `id2label` and `label2id` dictionaries as additional arguments. 

> How do I freeze the weights for training mask head for 25 epochs

```
for name, param in model.named_parameters():
  if name.startswith('detr'):
     param.requires_grad = False
```

> How do we edit the classifier layer of the model say by default this will have 92 class, but from the above example if I have only 2 class (balloon, 'N/A') how should I change them?

There's a new argument called `ignore_mismatched_sizes` which you can set to `True`. If you then specify a different number of labels, no error will be thrown (only a warning), as shown above.
<|||||>> If you want to do panoptic segmentation, you first need to load the model as follows:

Instead of using resnet50-panoptic how can I use my model from object detection (`DetrForObjectDetection` method) to train for panoptic segmentation<|||||>So for panoptic segmentation, DETR works as follows:

1) you first need to train a `DetrForObjectDetection` model to detect bounding boxes + classes (around both things + stuff classes). Let's say you have 10 classes in total (things + stuff), then you can initialize the model as follows:
```
from transformers import DetrForObjectDetection

# replace COCO classification head by custom one 
object_detection_model = DetrForObjectDetection.from_pretrained('facebook/detr-resnet-50', num_labels=10, ignore_mismatched_sizes=True)
# fine-tune this model on custom data
```

You've probably already done this, see my tutorial notebook: https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb
2) next, you can initialize a `DetrForSegmentation` model with the weights you obtained in step 1. This can be done as follows:
```
from transformers import DetrConfig, DetrForSegmentation

config = DetrConfig()
model = DetrForSegmentation(config)
# set the weights of the object detection model
model.detr = object_detection_model
```

This will give you a model that has all layers already initialized with some trained weights, except the mask head, which will be randomly initialized. 
3) next, you can freeze all layers except the mask head, and train for 25 epochs. Freezing can be done as follows:
```
for name, param in model.named_parameters():
  if name.startswith('detr'):
     param.requires_grad = False
```
<|||||>@NielsRogge Thanks for your reply.

I still can't figure out on adding the **53 COCO stuff** class to my custom data in Objectdetection. I am following the above finetune notebook which you have shared.

I have this doubt, should I download the COCO-17 val dataset and combine my custom data for the model to learn the stuff classes or just increase the class_emed layer from 100,4 (3+1 things class) to 100,57 (4+53). But in this case how to add this class to DetrConfig (id2class).

this is the notebook link :- [colab](https://colab.research.google.com/drive/1v1G2grxKrsnvVbJMMulY7xr4k9AwE5IF)
this custom data link:- [drive](https://drive.google.com/file/d/1ydE8KAojQk5HRfNG6GMLzkq-E_GeDOLr/view?usp=sharing)

One general question in your notebook the model is trained using `pl` rather that `torch` what is the reason for using `pl`

<|||||>If you want a neural network to learn additional classes, it's advised to add a new classification head and fine-tune the model on all classes you want. So indeed, now the class embedding layer should have 57 outputs.

> One general question in your notebook the model is trained using pl rather that torch what is the reason for using pl

Because it's very easy to train PyTorch models. You can of course just train using native PyTorch or using HuggingFace Accelerate, or using HuggingFace's Trainer, etc.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.