Body: <p>I want to get the Llama-2 rotary embeddings. I do <code>print(model)</code> and get the following output:
  <a href="https://i.sstatic.net/jFu2Z.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/jFu2Z.png" alt="enter image description here" /></a></p>
  <p>In the picture I highlight the rotary embeddings.</p>
  <p>How can get the rotary embeddings and how can I interpret the output? What means 32x LLamaDecoderLayer and in its round brakets are four layer plus LlamaRotaryEmbeddings?</p>
  <p>It's possible to get the embeddings as the first hidden-state <code>hidden_state[0]</code> and I want to know, which hidden-state represents the rotary embeddings.
  Am I right, that there are several rotary embeddings?</p>
  <p>Thanks in forward.</p>
  <p>Best regards.</p>
  
Tags: <neural-networks><natural-language-processing><transformer><word-embedding><large-language-models>
Title: How to get Llama-2 Rotary Embeddings?
UserDisplayName: Christian01
ThreadText: # How to get Llama-2 Rotary Embeddings?
  
  (Asked by: username_0 on 2023-09-29)
  
  I want to get the Llama-2 rotary embeddings. I do `print(model)` and get the following output:
  [![enter image description here](https://i.sstatic.net/jFu2Z.png)](https://i.sstatic.net/jFu2Z.png)
  
  In the picture I highlight the rotary embeddings.
  
  How can get the rotary embeddings and how can I interpret the output? What means 32x LLamaDecoderLayer and in its round brakets are four layer plus LlamaRotaryEmbeddings?
  
  It's possible to get the embeddings as the first hidden-state `hidden_state[0]` and I want to know, which hidden-state represents the rotary embeddings.
  Am I right, that there are several rotary embeddings?
  
  Thanks in forward.
  
  Best regards.
  
  **Tags:** neural-networks, natural-language-processing, transformer, word-embedding, large-language-models
  
  ---
  
  ## 0 answers
  
  