Body: <p>** As I am new to the field , the question might feel little abstract and naïve considering my experience.
  I am studying the Transformer architecture and trying to understand the various components within it. I am not able to understand the purpose of concatenating the attention head vectors, when the researchers could have used some other method to fuse the vectors. I wanted to understand the intuition behind that. I got a very good conversation on stack exchange itself (<a href="https://ai.stackexchange.com/questions/20948/combine-two-feature-vectors-for-a-correct-input-of-a-neural-network?newreg=666bdbcf8c0f4470ba6d740b00f4050f">Combine two feature vectors for a correct input of a neural network</a>). Taking the guidance from the link I wanted to understand whether the attention head outputs are linearly separable and that's why the researchers decided to concatenate the vectors? Or is it something related to direct vector sum for subspaces (not sure), or there is any proof relating to Information Theory wherein somebody has shown that the information loss in lesser that way (I am speculating). Please guide me through the concept.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-17T23:09:01.413Z
Comments.Text: _the attention head outputs are linearly separable_ this has nothing to do with concatenation... how would you combine them otherwise? you can choose to sum them, to weighted sum them, or to concatenate them, which is by far the most flexible, because the net will learn how to combine them
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-18T05:00:26.767Z
Comments.Text: @AlbertoSinigaglia make an answer
Comments.UserDisplayName: rjurney
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-18T05:09:30.917Z
Comments.Text: Thanks @AlbertoSinigaglia, but why only concatenation, what's the benefit of that compared to the ways you have provided.
Comments.UserDisplayName: Buddha Dev Bhattacharjee
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-18T21:37:44.383Z
Comments.Text: The other ones have some sort of biases, for example sum has the fact that the order is not relevant (1+2 = 2+1), however attention heads learn to attend to different concepts, so the invariance might not be a good idea
Comments.UserDisplayName: Alberto
Tags: <machine-learning><natural-language-processing><transformer><attention>
Title: Concatenation of Feature vectors in transformers before passing to fcnn
UserDisplayName: Buddha Dev Bhattacharjee
ThreadText: # Concatenation of Feature vectors in transformers before passing to fcnn
  
  (Asked by: username_0 on 2023-08-17)
  
  \*\* As I am new to the field , the question might feel little abstract and naïve considering my experience.
  I am studying the Transformer architecture and trying to understand the various components within it. I am not able to understand the purpose of concatenating the attention head vectors, when the researchers could have used some other method to fuse the vectors. I wanted to understand the intuition behind that. I got a very good conversation on stack exchange itself ([Combine two feature vectors for a correct input of a neural network](https://ai.stackexchange.com/questions/20948/combine-two-feature-vectors-for-a-correct-input-of-a-neural-network?newreg=666bdbcf8c0f4470ba6d740b00f4050f)). Taking the guidance from the link I wanted to understand whether the attention head outputs are linearly separable and that's why the researchers decided to concatenate the vectors? Or is it something related to direct vector sum for subspaces (not sure), or there is any proof relating to Information Theory wherein somebody has shown that the information loss in lesser that way (I am speculating). Please guide me through the concept.
  
  **Tags:** machine-learning, natural-language-processing, transformer, attention
  
  > **Comments:**
  > **username_1**: _the attention head outputs are linearly separable_ this has nothing to do with concatenation... how would you combine them otherwise? you can choose to sum them, to weighted sum them, or to concatenate them, which is by far the most flexible, because the net will learn how to combine them  
  > **username_2**: @AlbertoSinigaglia make an answer  
  > **username_0**: Thanks @AlbertoSinigaglia, but why only concatenation, what's the benefit of that compared to the ways you have provided.  
  > **username_1**: The other ones have some sort of biases, for example sum has the fact that the order is not relevant (1+2 = 2+1), however attention heads learn to attend to different concepts, so the invariance might not be a good idea  
  
  ---
  
  ## 0 answers
  
  