Answers.Body: <p>In image projection models like Llava image features are first passed through an image encoder (e.g., a CNN or ViT) and then projected into the shared text embedding space. Once projected, the image embeddings are treated as if they are a <em>separate</em> sequence of tokens alongside the text tokens. Each sequence (image and text) is processed by the transformer <em>independently</em>, and their interaction is restricted to how the sequences are interleaved and combined at the input stage either during training or inference.</p>
  <p>In contrast, cross-attention such as in Llama 3.2 means how each granular <em>token</em> (whether from the text or image) can attend to tokens from the other modality during each layer’s computation, which creates iterative interactions where both modalities can continuously inform and refine the model’s understanding. At lower layers this might be simple matching (e.g., &quot;dog&quot; in text and a &quot;dog&quot; in the image), while at higher layers this might involve more abstract reasoning (e.g., &quot;The dog in the park is barking at the bird&quot;). Also such iterative interactions can learn certain visual features which are only relevant in specific textual contexts, for instance, recognizing a dog in the image may not be helpful unless the context in the text also mentions a dog.</p>
  <p>Finally projection model's fixed transformation of an image into a text-based space via feature extraction could lose important spatial or semantic details of the image. In cross-attention models, the model preserves the image's richness using separate image tokens and maintaining their detailed information through the interaction with the text. This is especially beneficial in tasks that require fine-grained understanding, like image captioning, where knowing the exact position of objects and their relationships to text is critical.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-06T10:57:17.330Z
Answers.Text: Thank you. This part, about Llava "Each sequence (image and text) is processed by the transformer independently" -- are you sure? My understanding was that the image-projections and text-tokens-embeddings are interleaved, as part of one LLM input, and the training comes from backpropagating loss from the "Answer" part in the (Image, Qustion, Answer) triple...
Answers.UserDisplayName: Peter Franek
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-07T00:58:28.947Z
Answers.Text: Appreciate your clarification as here by *processed independently* I mean relative to cross attention layered processing, indeed, in Llava the two sequences of different modality are combined as a single sequence fed into LLM for language modeling loss (typically CE loss). But even with such (Image, Question, Answer) triple supervised learning, there's no layered granular interaction as the flexible cross attention. Hope this clarifies.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-06T02:39:16.437Z
Answers.LastActivityDate: 2024-11-07T01:03:07.470Z
Answers.LastEditDate: 2024-11-07T01:03:07.470Z
Answers.UserDisplayName: cinch
Body: <p>I would like to understand the internals of Meta's new multimodal Vision-Text 3.2 models. Not much I could find in online sources or blogs. The <a href="https://github.com/huggingface/transformers/tree/eab6c491d439e83d5e31c660df6f7e36592eb0a2/src/transformers/models/mllama" rel="nofollow noreferrer">code</a> is available and I'm trying to read it, but my understanding is still limited.</p>
  <p>It seems to me that the main ingredient is that we have an image and a text</p>
  <ul>
  <li>the image is encoded using a pretrained image encoder first, but I'm not sure if it was trained separately or as part of the overall Llama3.2 training</li>
  <li>Both &quot;image sequence&quot; and &quot;text sequence&quot; interact in some cross-attention layers, during the image forward pass</li>
  </ul>
  <p>I'm not sure how they handle multiple images interleaved with texts, however.</p>
  <p>Are there any helpful online sources with details, or would someone educated ellaborate on the details of the architecture and training here?</p>
  <p>I would also like to understand what is the advantage of this cross-attention compared to <a href="https://github.com/haotian-liu/LLaVA" rel="nofollow noreferrer">Llava</a> which only uses a projection of the encoded image sequence to the Llama's input embedding space and seems to be conceptually simpler.</p>
  <p>I will also appreciate partial answers or relevant remarks.</p>
  
Tags: <large-language-models><resource-request><vision-transformer>
Title: Insight into architecture details of Vision Llama 3.2
UserDisplayName: Peter Franek
ThreadText: # Insight into architecture details of Vision Llama 3.2
  
  (Asked by: username_0 on 2024-10-31)
  
  I would like to understand the internals of Meta's new multimodal Vision-Text 3.2 models. Not much I could find in online sources or blogs. The [code](https://github.com/huggingface/transformers/tree/eab6c491d439e83d5e31c660df6f7e36592eb0a2/src/transformers/models/mllama) is available and I'm trying to read it, but my understanding is still limited.
  
  It seems to me that the main ingredient is that we have an image and a text
  
  * the image is encoded using a pretrained image encoder first, but I'm not sure if it was trained separately or as part of the overall Llama3.2 training
  * Both "image sequence" and "text sequence" interact in some cross-attention layers, during the image forward pass
  
  I'm not sure how they handle multiple images interleaved with texts, however.
  
  Are there any helpful online sources with details, or would someone educated ellaborate on the details of the architecture and training here?
  
  I would also like to understand what is the advantage of this cross-attention compared to [Llava](https://github.com/haotian-liu/LLaVA) which only uses a projection of the encoded image sequence to the Llama's input embedding space and seems to be conceptually simpler.
  
  I will also appreciate partial answers or relevant remarks.
  
  **Tags:** large-language-models, resource-request, vision-transformer
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-06. Score: 2
  
  In image projection models like Llava image features are first passed through an image encoder (e.g., a CNN or ViT) and then projected into the shared text embedding space. Once projected, the image embeddings are treated as if they are a *separate* sequence of tokens alongside the text tokens. Each sequence (image and text) is processed by the transformer *independently*, and their interaction is restricted to how the sequences are interleaved and combined at the input stage either during training or inference.
  
  In contrast, cross-attention such as in Llama 3.2 means how each granular *token* (whether from the text or image) can attend to tokens from the other modality during each layer’s computation, which creates iterative interactions where both modalities can continuously inform and refine the model’s understanding. At lower layers this might be simple matching (e.g., "dog" in text and a "dog" in the image), while at higher layers this might involve more abstract reasoning (e.g., "The dog in the park is barking at the bird"). Also such iterative interactions can learn certain visual features which are only relevant in specific textual contexts, for instance, recognizing a dog in the image may not be helpful unless the context in the text also mentions a dog.
  
  Finally projection model's fixed transformation of an image into a text-based space via feature extraction could lose important spatial or semantic details of the image. In cross-attention models, the model preserves the image's richness using separate image tokens and maintaining their detailed information through the interaction with the text. This is especially beneficial in tasks that require fine-grained understanding, like image captioning, where knowing the exact position of objects and their relationships to text is critical.
  
  > **Comments:**
  > **username_0**: Thank you. This part, about Llava "Each sequence (image and text) is processed by the transformer independently" -- are you sure? My understanding was that the image-projections and text-tokens-embeddings are interleaved, as part of one LLM input, and the training comes from backpropagating loss from the "Answer" part in the (Image, Qustion, Answer) triple...  
  > **username_1**: Appreciate your clarification as here by *processed independently* I mean relative to cross attention layered processing, indeed, in Llava the two sequences of different modality are combined as a single sequence fed into LLM for language modeling loss (typically CE loss). But even with such (Image, Question, Answer) triple supervised learning, there's no layered granular interaction as the flexible cross attention. Hope this clarifies.  
  
  ---
  
  