BramVanroy:

“embeddings for each layer” but “output of embeddings” + “outputs of each layer”. So for a 12 layer model, you’d get the embedding output (1 tensor) + the outputs of the 12 following layers (12) = 13 tensors in total.


I’d also like to jump in as well because I can’t seem to find the answer I’m looking for anywhere else. Say that I want to extract speech vector embeddings with a Wav2Vec2 model and use them for a classification task. For that, I suppose I need the last hidden state of my model. If so, do I take tensor at the last index or the first index in the hidden_states tuple?
Intuitively, I think I should take one at the last index, as that would be the output of the last layer. However, I can’t be sure because I’m confused by what “output of embeddings/embedding output” means. (“Output of embeddings” doesn’t sound very self-explanatory. Is it supposed to be the initial embeddings of the speech input before going through model layers?)