Error projecting concatenated Fourier Features.
### System Info

- `transformers` version: 4.27.4
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.9.16
- Huggingface_hub version: 0.13.4
- PyTorch version (GPU?): 2.0.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers.models.perceiver.modeling_perceiver import PerceiverImagePreprocessor

input_preprocessor = PerceiverImagePreprocessor(
    config, 
    prep_type="conv",
    spatial_downsample=4,
    temporal_downsample=1,
    position_encoding_type="fourier",
    in_channels=4,
    out_channels=256,
    conv2d_use_batchnorm=True,
    concat_or_add_pos="concat",
    project_pos_dim=128,
    fourier_position_encoding_kwargs = dict(
        num_bands=64,
        max_resolution=[25,50], # 4x downsample include
    )
    )

test = torch.randn(1,4,100,200)
inputs, modality_sizes, inputs_without_pos = input_preprocessor(test)
```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[4], line 23
     21 test = torch.randn(1,4,100,200)
     22 # preprocessor outputs a tuple
---> 23 inputs, modality_sizes, inputs_without_pos = input_preprocessor(test)
     25 print(inputs.shape)
     26 print(inputs_without_pos.shape)

torch/nn/modules/module.py:1501), in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

transformers/models/perceiver/modeling_perceiver.py:3226), in PerceiverImagePreprocessor.forward(self, inputs, pos, network_input_is_1d)
   3223     else:
   3224         raise ValueError("Unsupported data format for conv1x1.")
-> 3226 inputs, inputs_without_pos = self._build_network_inputs(inputs, network_input_is_1d)
   3227 modality_sizes = None  # Size for each modality, only needed for multimodal
   3229 return inputs, modality_sizes, inputs_without_pos

transformers/models/perceiver/modeling_perceiver.py:3169), in PerceiverImagePreprocessor._build_network_inputs(self, inputs, network_input_is_1d)
   3166     pos_enc = self.position_embeddings(index_dims, batch_size, device=inputs.device, dtype=inputs.dtype)
   3168 # Optionally project them to a target dimension.
-> 3169 pos_enc = self.positions_projection(pos_enc)
   3171 if not network_input_is_1d:
   3172     # Reshape pos to match the input feature shape
   3173     # if the network takes non-1D inputs
   3174     sh = inputs.shape

torch/nn/modules/module.py:1501), in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

torch/nn/modules/linear.py:114), in Linear.forward(self, input)
    113 def forward(self, input: Tensor) -> Tensor:
--> 114     return F.linear(input, self.weight, self.bias)

RuntimeError: mat1 and mat2 shapes cannot be multiplied (1176x258 and 256x128)

### Expected behavior

Linear projection input number of features should equal the number of positional features.

When concatenating Fourier Features the expected number of positional features is: (2 * num_bands * num_dims) + 2.
The build_position_encoding() function takes as input the out_channels which is used to define the input number of features for linear projection. PerceiverImagePreprocessor incorrectly passes in the embedding dimension to out_channels for positional encoding projection.

Possible Fix:
Use the positional encoding class method output_size() to pull number of input features for projection within build_position_encoding() function.

positions_projection = nn.Linear(output_pos_enc.output_size(), project_pos_dim) if project_pos_dim > 0 else nn.Identity()
@sr-ndai Thanks for raising this issue and detailed reproduction. 

Could you share the config or checkpoint being used in this example? <|||||>@amyeroberts No problem, here is the configuration I used:
```python
from transformers import PerceiverConfig

config = PerceiverConfig(
    num_latents = 128,
    d_latents=256,
    num_blocks=6,
    qk_channels = 256,
    num_self_attends_per_block=2,
    num_self_attention_heads=8,
    num_cross_attention_heads=8,
    self_attention_widening_factor=2,
    cross_attention_widening_factor=1,
    hidden_act='gelu_new',
    attention_probs_dropout_prob=0.1,
    use_query_residual=True,
    num_labels = 128 # For decoder
)
```