Quick question difference output of Bert models compared to Electra
Hi everyone,

I am a little confused at the moment.

When I run `outputs = model.roberta(input_ids, attention_mask)` or `model.albert(input_ids, attention_mask)` the length of output is 3 and looks like this:

`output[0].shape = [4, 249, 768]`
`output[1][0].shape = [768]`
`output[2][0].shape = [4,249,768]`

When I run model.electra(input_ids, attention_mask)` the length of output is 2 and looks like this:

`output[0].shape = [4, 251, 768]`
`output[1][0].shape = [4, 251, 768]`

I checked the config files of both models and both `output hidden states` etc. seems to be set to False and in the code I don't specify to output anything extra for either model.

Can someone explain why Electra all of a sudden outputs less compared to other models and also what output[0], output[1] and output[2] mean for Bert and for Electra?

I checked the documentation but there it states all the output except scores is optional so I am confused what output contains know since to my understanding I haven't specified to output and of the optional output.

Thanks in advance for helping me clear this confusion up.

For questions like this it's often best to dive into the source code. It helps understanding it a lot.

But your example seems wrong, you may have made some copy-pasting errors. If `output[1][0].shape = [768]` then also, because tensors across an axis must have same dimensions (like matrices) `output[2][0].shape = [768]`.<|||||>@BramVanroy I double checked but it is not a copy-pasting error as far as I can tell. I will dive into the source code to try to understand why the length of electra's output is different from the bert models.<|||||>Can you post a reproducible full, but minimal, example?<|||||>@BramVanroy I figured it out. Turns out even though in the config the output_hidden_states is set to false somewhere hidden in the code I am using it get sets to true.

In case of the bert models output[2] are the hidden layers and for electra [1] are the hidden layers. I'm still not very sure what output[1] is in case of the output of the bert models but for my particular use case it is not important right now.

Thank you for taking your time to help me. I will close the issue.