How to get the last 4 Hidden states from the feature extraction pipeline
I have defined a pipeline for Feature extraction
```
# Create the pipeline
p = pipeline(
    task="feature-extraction",
    tokenizer="microsoft/biogpt",
    model="microsoft/biogpt",
    framework="pt",
    device=0
)
bio_gpt = AutoModel.from_pretrained("microsoft/biogpt", output_hidden_states= True)
bio_gpt = bio_gpt.to(device)
```

and I want to extract the embeddings of the last token of the last hidden state, and the Average Pooling of the last 4 layers using the pipeline approach I am doing it like this

_Last token of the last hidden state:_

```
def extract_last_token(last_hidden_states):
    last_hidden_states = np.array(last_hidden_states)
    return last_hidden_states[:,-1,:]

# Process the data using the pipeline
results = p([row["text"] for _, row in df2.iterrows()])

# Extract the last token of the last hidden state
embeddings = [extract_last_token(hidden_state) for hidden_state in results]

# Create a DataFrame to store the results
df2["embeddings2"] = embeddings
```
_Average pooling of the last 4 layers:_
```
def mean_pooling(last_hidden_states, ):
    last_4_layers = last_hidden_states[-4:]  # Consider the last 4 layers
    return np.mean(last_4_layers, axis=1)

# Process the data using the pipeline
results = p([row["text"] for _, row in df2.iterrows()])

features = np.squeeze(results)

print(features.shape)
# Perform mean pooling on the last hidden states
embeddings = [mean_pooling(hidden_state) for hidden_state in results]

# Create a DataFrame to store the results
df2["embeddings4"] = embeddings
```
The issues are:

1. When I extract the embeddings of the 4 last layers or the 12 last layers the embeddings are always the same

![image](https://github.com/huggingface/transformers/assets/138615931/70c265aa-4182-4265-bb22-ddc197388c03)

2. The embeddings of the last token of the last hidden state are different from the same embeddings using the "manual" method

![image](https://github.com/huggingface/transformers/assets/138615931/a7b9b629-9c65-4b76-b669-89d9e64103de)

Weardly in the above picture the 2 of the embeddings are the same but opposite row ids, this indicates another problem I don't see it if you can spot this I appreciate it.

Here is the code of how I did the manual version
```
output = bio_gpt(**model_inputs)

# Get the last state
last_state = output.last_hidden_state

cls_embeddings = last_state[:, -1, :]

# Print the last state
print(cls_embeddings)

# Assign cls_embeddings to "embeddings4" column in df2
df2["embeddings_manual"] = [cls_embeddings[i].cpu().detach().numpy() for i in range(len(df2))]
```
Hi, could you also provide the data `df2` (or another version of it if privacy is concerned).

Thanks.<|||||>> Hi, could you also provide the data `df2` (or another version of it if privacy is concerned).
> 
> Thanks.

Sure, its just a text and the label
![image](https://github.com/huggingface/transformers/assets/138615931/85f312c0-5112-4b7b-8c29-36eec2545a34)




<|||||>@Luke-4 

Not as an image please. Make it something that can be used to run the code snippet directly 🙏 
<|||||>> @Luke-4
> 
> Not as an image please. Make it something that can be used to run the code snippet directly 🙏

here hope this works:
https://drive.google.com/drive/folders/186rEP0ZMYc3tjR_EKBYNhYEx9sUtmSTj?usp=sharing




<|||||>I haven't look in full details. However, your input to `mean_pooling` (i.e. each element of the output from `p`) seems to have `[batch_dim, seq_len, hideen_dim]`. The `batch_dim` here is just `1`

When you do `last_hidden_states[-4:]` inside `mean_pooling`, it is actually the same element as `last_hidden_states`, as your are taking the last 4 elements along batch dimension (and not the different layers!). When you do `np.mean(..., axis=1)`, it's actually mean along the sequence dimension, and get a shape of `[batch_dim=1, hidden_dim]`.

This doesn't corresponds to what you describe that you want to get mean along the last 4 layers.
However, I am not sure if the `feature extraction pipeline` allow to get all hidden states (from all layers) - probably yes.

Could you verify first, please?



```python
r = p(["I love dog", "I love cat too", "I love cat that meow meow a lot"])

import numpy as np

print(len(a))

print(len(a[0]))
print(len(a[1]))
print(len(a[2]))

print(len(a[0][0]))
print(len(a[1][0]))
print(len(a[2][0]))

print(np.array(a[0][0]).shape)
print(np.array(a[1][0]).shape)
print(np.array(a[2][0]).shape)
```
gives
```bash
3
1
1
1
5
5
6
(5, 1024)
(5, 1024)
(12, 1024)
```