Self-Attention Layers for Perceiver Decoder
This feature request is about the PerceiverIO codebase, specifically about the implemented decoders. It seems like there are no self-attention layers, only a single cross-attention layer.  Is that by design?  Several current published methods mention cross- and self-attention layers as part of the decoding process, would it be possible to include this functionality?   

https://github.com/huggingface/transformers/blob/7799b6128feb17b63c47ed77a71fa367d26492d2/src/transformers/models/perceiver/modeling_perceiver.py#L2035
Hi,

> It seems like there are no self-attention layers, only a single cross-attention layer. Is that by design?

Yes, the authors used only a single cross-attention layer. 

If you want, you can fork the library and tweak `modeling_perceiver.py` to include additional self-attention layers in the decoder.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>Closing this issue, as I believe I've answered your question. Feel free to re-open if you have further questions.