What it means is not “embeddings for each layer” but “output of embeddings” + “outputs of each layer”. So for a 12 layer model, you’d get the embedding output (1 tensor) + the outputs of the 12 following layers (12) = 13 tensors in total.
You can get the attention values back (output_attentions=True), but AFAIK not the query vectors.