Why couldn't I use encoder_hidden_states when position_ids is not None? GPT2Model.foward()
`device` is required in GPT2Model.foward() if I'd like to use encoder_hidden_states.
```
          if self.config.add_cross_attention and encoder_hidden_states is not None:
              encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.size()
              encoder_hidden_shape = (encoder_batch_size, encoder_sequence_length)
              if encoder_attention_mask is None:
                  encoder_attention_mask = torch.ones(encoder_hidden_shape, device=device)
              encoder_attention_mask = self.invert_attention_mask(encoder_attention_mask)
          else:
              encoder_attention_mask = None
  ```
https://github.com/huggingface/transformers/blob/26212c14e5570aff40b90c11495d97dada4272fb/src/transformers/models/gpt2/modeling_gpt2.py#L682

But the only place that sets `device` is in another *if statement*.
```python
        if position_ids is None:
            device = input_ids.device if input_ids is not None else inputs_embeds.device
            position_ids = torch.arange(past_length, input_shape[-1] + past_length, dtype=torch.long, device=device)
            position_ids = position_ids.unsqueeze(0).view(-1, input_shape[-1])
```
https://github.com/huggingface/transformers/blob/26212c14e5570aff40b90c11495d97dada4272fb/src/transformers/models/gpt2/modeling_gpt2.py#L653

And I was wondering why is it required to have 'position_ids==None' when I just want to use `encoder_hidden_states` .
Am I missing something?

I ran into this problem when trying to use GPT2LMHeadModel for image captioning tasks.
Ah, I think this is an issue indeed, the device statement shouldn't be inside that `if` statement. Do you want to open a PR to fix this?<|||||>Sure. <|||||>Fixed by #11292 