How to get pooler state's (corresponds to CLS token) attention vector?
The following model definition returns the attention vector for tokens corresponding to the input sequence length, i.e. ```x.size(1)```. How do I procure the attention vector of the pooler state (output embedding corresponding to the CLS token)?

```Python
model = BertModel.from_pretrained('bert-base-uncased', output_attentions=True)
outputs = self.model(x, attention_mask = x_mask)
last_layer_attentions = outputs[2][-1] # [batch_size, num_heads, x.size(1), x.size(1)]
# I want the attention vector for pooler state, i.e. [batch_size, num_heads,  1, x.size(1)]
```

Hi, the pooler takes as input the last layer hidden-state of the first token of the sentence (the `[CLS]` token). So the attention used to compute the pooler input is just the attention for this token.<|||||>@thomwolf If my understanding is right, the last layer's attention vector should be of size ```[batch_size, num_heads, (x.size(1)+1), (x.size(1)+1)]```, corresponding to the **[CLS]** embedding and x.size(1) token embeddings. However, ```output[2][-1]``` only returns ```[batch_size, num_heads, x.size(1), x.size(1)]``` dimensional attention map, which, I am guessing, corresponding to the input sequence (**x.size(1)**) and not the **[CLS]** token.

How do I get the attention vector corresponding to the **[CLS]** token? Also, can you mention which of the two **x.size(1)** axes corresponds to the input layer and the output layer?