Possibly a bug in Pix2Struct outputs
I'm sorry if I'm wrong, as I don't have much experience with transformers internals.
I was playing with Pix2Struct and trying to visualise attention on input image. `output.cross_attentions` shape didn't make much sense as it didn't have `patch_count` as any of dimensions. After inspecting `modeling_pix2struct.py` I have notices the following
```
            # layer_outputs = hidden-states, key-value-states (self-attention position bias), (self-attention weights),
            # (cross-attention position bias), (cross-attention weights)
```
And then later
```
            if output_attentions:
                all_attentions = all_attentions + (layer_outputs[2],)
                all_cross_attentions = all_cross_attentions + (layer_outputs[3],)
```

As I understand `layer_outputs[3]` is `(self-attention weights)` and it should be replaces with `layer_outputs[5]` which is `(cross-attention weights)`. The same goes of `(layer_outputs[2],) => (layer_outputs[3],)`

Does it make sense, or am I getting something wrong?

I tried to patch it locally and output + visualisation make sense (highlight image patch with information in token)

https://github.com/huggingface/transformers/blob/fe861e578f50dc9c06de33cd361d2f625017e624/src/transformers/models/pix2struct/modeling_pix2struct.py#L1550C74-L1550C74
Hi @artyomxyz, thanks for reporting this issue.

Yes, you're right, there is a current issue with indexing in the Pix2Struct model. Related PR: #23985

cc @younesbelkada <|||||>> Hi @loveisp ! Again thanks for your contribution on this Can you share with us why this PR got closed? The PR should also fix #24717 so it would be great to merge it :D

I initially made the change like this, but it didn't pass all the tests. The len(layer_outputs) here seemed a bit strange, so I changed it to what came later. Even though it can pass all the tests, there are still issues with the underlying logic. Regarding this matter, you can take a look at my discussion with @amyeroberts . I realized that I cannot fix this bug in a short amount of time, so I closed it. Will it be sufficient to make this change as he suggested, so that it passes all the tests? If so, then go ahead and make the change.