Answers.Body: <p>There are various methods to combine two sets of features (say <span class="math-container">$x$</span> and <span class="math-container">$y$</span>):</p>
  <ul>
  <li><strong>Concatenation</strong> (as done in figure): done along a given dimension (usually last axis), to obtain new features <span class="math-container">$z=[x\ y]$</span>. The resulting <span class="math-container">$z$</span>, will have a dimension that is <span class="math-container">$dim(z) = dim(x)+dim(y)$</span>. The advantage of this method is that is <em>flexible</em>, meaning the next layers should understand how to make good use of the extra channels or dimensions. Disadvantage: the result is larger (more dimensions).</li>
  <li><strong>Addition</strong>: just element-wise add the two features <span class="math-container">$z=x+y$</span>. Advantage: same num. of dimensions, but less flexible.</li>
  </ul>
  <ul>
  <li><strong>Multiplication</strong>: element-wise multiply the two features <span class="math-container">$z=x\odot y$</span>. Same pros and cons of the addition strategy.</li>
  </ul>
  <p>Now, it's not possible a priori to determine which of these is better. You should just try each of them and see how it performs.</p>
  <p>A <strong>weighted sum</strong> can be a good approach especially if <span class="math-container">$x$</span> and <span class="math-container">$y$</span> have some meaning:</p>
  <ul>
  <li><span class="math-container">$z = a\cdot x + b\cdot y$</span>. The coefficients <span class="math-container">$a$</span> and <span class="math-container">$b$</span> can be set manually, learned (1-dimensional scaling factors, or have the same dim. of the features), or set like <span class="math-container">$b=(1-a)$</span>.</li>
  <li>If you learn <span class="math-container">$a$</span> and <span class="math-container">$b$</span> you may want to bound them in [0,1] via <code>sigmoid</code> or <code>softmax</code> (if multi-dimensional): this is known as <em>gating</em>, if I remember well.</li>
  <li>In principle you can even combine addition with multiplication.</li>
  </ul>
  <p>An even more sophisticated approach would be to have a small sub-network with two inputs and one output, to learn how to best combine <span class="math-container">$x$</span> and <span class="math-container">$y$</span> in a non-linear way.</p>
  <p>The last tip is that if you have some domain knowledge on the problem and data, you may be able to introduce that when designing the operation that combines the two feature sets.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-18T20:16:02.487Z
Answers.Text: Anyalone Thanks for the inputs. I think I will stay away from parameters that I will have to further learn in the model. Because the goal is to achieve min possible training time. So, less the parameters, the better.
Answers.UserDisplayName: programmer_04_03
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-18T20:26:35.503Z
Answers.Text: And I was also thinking about some convolution operations with predefined kernels such as Random Gaussian or average value. Do you think that will be helpful?
Answers.UserDisplayName: programmer_04_03
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-19T09:10:50.923Z
Answers.Text: In general, when you want to combine two feature sets you want to also combine (and so preserve) their respective information. I guess introducing random kernels is a bad idea, maybe the average is a little better. Anyway, in the paper section D (page 4) they propose 2 strategies: *addition* and *channel* strategy. Have you looked at the channel strategy? Maybe you can combine (or weight) the RGB features with the pooled ones of the multispectral input
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-18T20:01:48.163Z
Answers.LastActivityDate: 2023-04-18T20:01:48.163Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I am working image reconstruction project. It is a part of multispectral image fusion. I am referring paper in the link mentioned below.</p>
  <p>paper link: <a href="https://arxiv.org/pdf/2101.09643v1.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2101.09643v1.pdf</a></p>
  <p>For image reconstruction, authors have used a network which looks like this:
  <a href="https://i.sstatic.net/zpVRt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zpVRt.png" alt="enter image description here" /></a></p>
  <p>And after implementing this, for training, and changing parameters, the loss (difference between reconstructed and actual image) is calculated.</p>
  <p>In the image above, after extraction of two feature sets, authors are simply giving these feature sets to the decoder network after concatenation. I was wondering, can there be any better way than just concatenating these two feature sets, like weighted sum maybe ? For combining these feature sets?</p>
  <p>I will really appreciate if anyone can suggest any ideas.</p>
  
Tags: <deep-learning><image-processing><image-generation>
Title: Methods for combining features other than concatenation
UserDisplayName: programmer_04_03
ThreadText: # Methods for combining features other than concatenation
  
  (Asked by: username_0 on 2023-04-18)
  
  I am working image reconstruction project. It is a part of multispectral image fusion. I am referring paper in the link mentioned below.
  
  paper link: <https://arxiv.org/pdf/2101.09643v1.pdf>
  
  For image reconstruction, authors have used a network which looks like this:
  [![enter image description here](https://i.sstatic.net/zpVRt.png)](https://i.sstatic.net/zpVRt.png)
  
  And after implementing this, for training, and changing parameters, the loss (difference between reconstructed and actual image) is calculated.
  
  In the image above, after extraction of two feature sets, authors are simply giving these feature sets to the decoder network after concatenation. I was wondering, can there be any better way than just concatenating these two feature sets, like weighted sum maybe ? For combining these feature sets?
  
  I will really appreciate if anyone can suggest any ideas.
  
  **Tags:** deep-learning, image-processing, image-generation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-04-18. Score: 1
  
  There are various methods to combine two sets of features (say $x$ and $y$):
  
  * **Concatenation** (as done in figure): done along a given dimension (usually last axis), to obtain new features $z=[x\ y]$. The resulting $z$, will have a dimension that is $dim(z) = dim(x)+dim(y)$. The advantage of this method is that is *flexible*, meaning the next layers should understand how to make good use of the extra channels or dimensions. Disadvantage: the result is larger (more dimensions).
  * **Addition**: just element-wise add the two features $z=x+y$. Advantage: same num. of dimensions, but less flexible.
  
  * **Multiplication**: element-wise multiply the two features $z=x\odot y$. Same pros and cons of the addition strategy.
  
  Now, it's not possible a priori to determine which of these is better. You should just try each of them and see how it performs.
  
  A **weighted sum** can be a good approach especially if $x$ and $y$ have some meaning:
  
  * $z = a\cdot x + b\cdot y$. The coefficients $a$ and $b$ can be set manually, learned (1-dimensional scaling factors, or have the same dim. of the features), or set like $b=(1-a)$.
  * If you learn $a$ and $b$ you may want to bound them in [0,1] via `sigmoid` or `softmax` (if multi-dimensional): this is known as *gating*, if I remember well.
  * In principle you can even combine addition with multiplication.
  
  An even more sophisticated approach would be to have a small sub-network with two inputs and one output, to learn how to best combine $x$ and $y$ in a non-linear way.
  
  The last tip is that if you have some domain knowledge on the problem and data, you may be able to introduce that when designing the operation that combines the two feature sets.
  
  > **Comments:**
  > **username_0**: Anyalone Thanks for the inputs. I think I will stay away from parameters that I will have to further learn in the model. Because the goal is to achieve min possible training time. So, less the parameters, the better.  
  > **username_0**: And I was also thinking about some convolution operations with predefined kernels such as Random Gaussian or average value. Do you think that will be helpful?  
  > **username_1**: In general, when you want to combine two feature sets you want to also combine (and so preserve) their respective information. I guess introducing random kernels is a bad idea, maybe the average is a little better. Anyway, in the paper section D (page 4) they propose 2 strategies: *addition* and *channel* strategy. Have you looked at the channel strategy? Maybe you can combine (or weight) the RGB features with the pooled ones of the multispectral input  
  
  ---
  
  