Answers.Body: <p><strong>Is there maybe a better way [to combine them into one vector in such a way that their order does not matter]?</strong></p>
  <p>There are many ways to perform feature fusion (see <a href="https://www.nature.com/articles/s41746-020-00341-z" rel="nofollow noreferrer">ref</a>).  The way you described is one way of doing it.  However, for most simple cases where feature fusion is for the type of data you are describing (i.e., tabular data), it is more common to <code>concatenate</code> the features at the input level.  The order of concatenation does not matter to the neural network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-29T13:41:34.543Z
Answers.Text: Could you elaborate on how for a neural network, the order of concatenation does not matter? From the same vectors, depending on the order of concatenation we will get completely different inputs to the network. How could it not matter?
Answers.UserDisplayName: Kasia
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-29T13:55:41.897Z
Answers.Text: Each of the input features feed into each one of the neurons in the first hidden layer.  The inputs are, therefore, all treated the same.  Thus, the order of concatenation doesn't matter.  This [diagram](https://preview.redd.it/pjj39fhl1pc61.png?width=640&format=png&auto=webp&s=37fd1d99b4bf9aa2ff64975ba64daa52a3a141f4) shows how that is true.
Answers.UserDisplayName: Snehal Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-30T01:08:13.793Z
Answers.Text: I disagree that concatenation would be invariant to the order of the inputs. In the given example the values of `h_1` will change if you permute the inputs, assuming that is weights are distinct for each input.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-30T01:34:37.437Z
Answers.Text: The important thing is to be consistent about the order in which the inputs are presented to the model, not the absolute ordering itself.  Think about it this way:  If you are feeding a neural network two features - petal length and sepal length for example - to predict the iris species, wouldn't it be a bizarre property of the model if it mattered which feature was in column 1 and which was in column 2 of your datatable?  Regardless, the order invariance is a directly related to the commutative property of addition.  w1*input1 + w2*input2 + b1 = w2*input2 + w1*input1 + b1.
Answers.UserDisplayName: Snehal Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-30T01:40:11.067Z
Answers.Text: You are right, @NikoNyrh. You cannot change the order randomly.  Once you have decided that feature 1 is feeding into input neuron 1, you must be consistent.  My point though is that the initial ordering of which feature you decide to be feature 1 which which one you decide to be feature 2 and so on is not considered by the model.
Answers.UserDisplayName: Snehal Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-29T13:17:08.107Z
Answers.LastActivityDate: 2022-10-29T13:25:33.710Z
Answers.LastEditDate: 2022-10-29T13:25:33.710Z
Answers.UserDisplayName: Snehal Patel
Answers.Body: <blockquote>
  <p>My first idea was to run each vector through the same activated linear layer and sum the results.</p>
  </blockquote>
  <p>That was my first thought as well, but actually it doesn't need to be a single layer but instead it can be a more complex neural network. Basically it will learn an embedding <span class="math-container">$R^n \rightarrow R^m$</span>, and you can use the sum (or average) of such embeddings to represent the set of your features. I wouldn't use any activation in the final layer, or maybe at most <code>tanh</code> to preserve &quot;symmetry&quot; between negative and positive values. Or who knows, maybe <code>ReLU</code> would work in you problem as well.</p>
  <p>I think average of the embeddings would be better than a plain sum, in case the number of &quot;active&quot; features may change and absent features have an embedding of all zeros. Or you normalize the sum so that the L1 or L2 norm is 1.</p>
  <p>Come to think of it, the <span class="math-container">$R^n \rightarrow R^m$</span> can be implemented as a stack of 1D convolution with &quot;kernel size&quot; and &quot;stride&quot; of one (Keras terms). Or it can be a &quot;stand-alone&quot; model of which outputs you sum together.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-01T15:36:53.727Z
Answers.LastActivityDate: 2022-11-01T15:36:53.727Z
Answers.UserDisplayName: NikoNyrh
Body: <p>My input to the model is a set of features that I encode in the form of five vectors of the same size consisting only of 0 and 1. I now want to combine them into one vector in such a way that their order does not matter.</p>
  <p>My first idea was to run each vector through the same activated linear layer and sum the results.</p>
  <p>Is there maybe a better way?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-31T14:04:02.990Z
Comments.Text: Could you give us a concrete example? And what do you mean by "the order of features does not matter", is it that if you permute their order the network's output should be unchanged?
Comments.UserDisplayName: NikoNyrh
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-31T20:58:59.910Z
Comments.Text: @NikoNyrh Simply put, I have to encode a set of sets of features. By set I mean an unordered collection of unique elements. I can encode each inner set as a vector of zeros and ones. Now I need a function that will turn the outer set into a single vector. Simply adding or multiplying these vectors seems to me not to be the right way.
Comments.UserDisplayName: Kasia
Tags: <deep-learning><linear-algebra>
Title: Encoding data in which the order of features does not matter
UserDisplayName: Kasia
ThreadText: # Encoding data in which the order of features does not matter
  
  (Asked by: username_0 on 2022-10-29)
  
  My input to the model is a set of features that I encode in the form of five vectors of the same size consisting only of 0 and 1. I now want to combine them into one vector in such a way that their order does not matter.
  
  My first idea was to run each vector through the same activated linear layer and sum the results.
  
  Is there maybe a better way?
  
  **Tags:** deep-learning, linear-algebra
  > **Comments:**
  > **username_1**: Could you give us a concrete example? And what do you mean by "the order of features does not matter", is it that if you permute their order the network's output should be unchanged?  
  > **username_0**: @NikoNyrh Simply put, I have to encode a set of sets of features. By set I mean an unordered collection of unique elements. I can encode each inner set as a vector of zeros and ones. Now I need a function that will turn the outer set into a single vector. Simply adding or multiplying these vectors seems to me not to be the right way.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-10-29. Score: 1
  
  **Is there maybe a better way [to combine them into one vector in such a way that their order does not matter]?**
  
  There are many ways to perform feature fusion (see [ref](https://www.nature.com/articles/s41746-020-00341-z)). The way you described is one way of doing it. However, for most simple cases where feature fusion is for the type of data you are describing (i.e., tabular data), it is more common to `concatenate` the features at the input level. The order of concatenation does not matter to the neural network.
  
  > **Comments:**
  > **username_0**: Could you elaborate on how for a neural network, the order of concatenation does not matter? From the same vectors, depending on the order of concatenation we will get completely different inputs to the network. How could it not matter?  
  > **username_2**: Each of the input features feed into each one of the neurons in the first hidden layer.  The inputs are, therefore, all treated the same.  Thus, the order of concatenation doesn't matter.  This [diagram](https://preview.redd.it/pjj39fhl1pc61.png?width=640&format=png&auto=webp&s=37fd1d99b4bf9aa2ff64975ba64daa52a3a141f4) shows how that is true.  
  > **username_1**: I disagree that concatenation would be invariant to the order of the inputs. In the given example the values of `h_1` will change if you permute the inputs, assuming that is weights are distinct for each input.  
  > **username_2**: The important thing is to be consistent about the order in which the inputs are presented to the model, not the absolute ordering itself.  Think about it this way:  If you are feeding a neural network two features - petal length and sepal length for example - to predict the iris species, wouldn't it be a bizarre property of the model if it mattered which feature was in column 1 and which was in column 2 of your datatable?  Regardless, the order invariance is a directly related to the commutative property of addition.  w1*input1 + w2*input2 + b1 = w2*input2 + w1*input1 + b1.  
  > **username_2**: You are right, @NikoNyrh. You cannot change the order randomly.  Once you have decided that feature 1 is feeding into input neuron 1, you must be consistent.  My point though is that the initial ordering of which feature you decide to be feature 1 which which one you decide to be feature 2 and so on is not considered by the model.  
  
  ---
  
  ### Answer by username_1 on 2022-11-01. Score: 0
  
  > My first idea was to run each vector through the same activated linear layer and sum the results.
  
  That was my first thought as well, but actually it doesn't need to be a single layer but instead it can be a more complex neural network. Basically it will learn an embedding $R^n \rightarrow R^m$, and you can use the sum (or average) of such embeddings to represent the set of your features. I wouldn't use any activation in the final layer, or maybe at most `tanh` to preserve "symmetry" between negative and positive values. Or who knows, maybe `ReLU` would work in you problem as well.
  
  I think average of the embeddings would be better than a plain sum, in case the number of "active" features may change and absent features have an embedding of all zeros. Or you normalize the sum so that the L1 or L2 norm is 1.
  
  Come to think of it, the $R^n \rightarrow R^m$ can be implemented as a stack of 1D convolution with "kernel size" and "stride" of one (Keras terms). Or it can be a "stand-alone" model of which outputs you sum together.
  
  
  ---
  
  