Thanks for your reply Bram!
I realized that my tired brain got confused between the outputs of past_key_values and hidden_states and I was actually asking about the len 2 tuple returned by each layer of the past_key_values. It seems like index 0 is key and index 1 is the value? But that this has nothing to do with the embedding.

past_key_values ( tuple(tuple(torch.FloatTensor)) , optional, returned when use_cache=True is passed or when config.use_cache=True ) –
Tuple of tuple(torch.FloatTensor) of length config.n_layers , with each tuple having 2 tensors of shape (batch_size, num_heads, sequence_length, embed_size_per_head) ) and optionally if config.is_encoder_decoder=True 2 additional tensors of shape (batch_size, num_heads, encoder_sequence_length, embed_size_per_head) .
Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values input) to speed up sequential decoding.

And sounds like I will have to invert the attention values and use the keys to work out the query vectors myself.
Thanks again and sorry for my confusion.