Answers.Body: <p><em><span class="math-container">$^*$</span>Note - Question is bit unclear, in case the answer doesn't addresses the question, please ask for edit/delete Request.</em></p>
  
  <hr>
  
  <h2>GENERALIZATION</h2>
  
  <p>Suppose there are multiple datasets denoted by <span class="math-container">$A_i$</span>. Datasets contain a set of Vectors <span class="math-container">$x_{j} $</span>. Mathematically <span class="math-container">$A_i = \{ x_j\}_{j=0}^n$</span>. We've to find an estimator function <span class="math-container">$\hat f$</span>, such that <span class="math-container">$\hat f( \vec r) = y, \, \vec r \in X$</span> where <span class="math-container">$X $</span> is a special dataset created by combining all <span class="math-container">$A_i$</span> which helps in classification into <span class="math-container">$y \in Y$</span> which is the set of classes.</p>
  
  <p>.</p>
  
  <p>As @Amir Mentioned out, linearly separable feature can be easily separated by straight combination of vectors i.e. if <span class="math-container">$x_u \in A_i, w_v \in A_j \dots$</span>, then <span class="math-container">$r = [x_1 \,x_2 \, \dots  \, x_u \, w_1 \, \dots w_v \dots]$</span>. Where, <span class="math-container">$r \in X$</span> which is the required dataset.</p>
  
  <p><em>There are cases where the features are not linearly separable</em>, We use <strong>basis expansion methods</strong><a href="https://towardsdatascience.com/non-linear-regression-basis-expansion-polynomials-splines-2d7adb2cc226" rel="nofollow noreferrer">[1]</a> to make required shape of hyperplane to separate the features. We create a new dataset combining <span class="math-container">$A_i \, \forall i \in C \subset \mathbb N$</span>. Suppose that the new dataset is <span class="math-container">$X$</span>, then <span class="math-container">$r \in X$</span> and <span class="math-container">$r = [r_0, r_1, \dots r_n].$</span></p>
  
  <p>Then,</p>
  
  <p><span class="math-container">$$r_1 = u_1^2v_1^2 \\ r_2 = \sin(u_2)\sin(v_2) \\ r_3 = ae^{u_3 + v_3} \\ r_4 = a v_4 v_4 + a_2 u_4^2 v_4^2 + \dots \\ \dots$$</span></p>
  
  <p>Here <span class="math-container">$u_p \in A_i; \, v_q \in A_j$</span></p>
  
  <p>Here you can use all the creativity to set <span class="math-container">$r = [r_1, r_2, \dots , r_n]$</span> and make a new dataset. What equations and what functions you chose fully depends on the kind of hyperplane shape you want to obtain. Basis expansion is just one of the methods for feature extraction is certainly one of the most flexible too. </p>
  
  <p>Now, you feed the newly created vectors into your trained estimator functions (which is Neural Net) which can classify things much easily now.</p>
  
  <p><em>In case of Regression/Classification without Neural Net needs some extra treatment to train the model</em><a href="https://medium.com/datadriveninvestor/improve-your-classification-models-using-mean-target-encoding-a3d573df31e8" rel="nofollow noreferrer">[2]</a>.</p>
  
  <hr>
  
  <p><strong><a href="https://medium.com/datadriveninvestor/improve-your-classification-models-using-mean-target-encoding-a3d573df31e8" rel="nofollow noreferrer">[2]</a>Note:</strong> There is also a big role of encoding. For example, if you encode colors by numbers <span class="math-container">$1, 2, 3$</span> for RGB or <span class="math-container">$10,01, 11$</span> fully changes everything and your features too. In such cases, You may even need different equations to make your required dataset <span class="math-container">$X$</span> and vectors <span class="math-container">$r$</span>.</p>
  
  <hr>
  
  <h2>REFERENCES:</h2>
  
  <ol>
  <li>Oleszak, Michal. <a href="https://towardsdatascience.com" rel="nofollow noreferrer">https://towardsdatascience.com</a>. <a href="https://towardsdatascience.com/non-linear-regression-basis-expansion-polynomials-splines-2d7adb2cc226" rel="nofollow noreferrer"><em>Non-linear regression: basis expansion, polynomials &amp; splines</em></a>. Sep 30, 2019. Web. 6 May 2020.</li>
  <li>Sangarshanan. <a href="https://medium.com" rel="nofollow noreferrer">https://medium.com</a>. <a href="https://medium.com/datadriveninvestor/improve-your-classification-models-using-mean-target-encoding-a3d573df31e8" rel="nofollow noreferrer"><em>Improve your classification models using Mean /Target Encoding</em></a>. Jun 23, 2018. Web. 6 May 2020.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T06:32:02.473Z
Answers.Text: I really appreciate your help, your answer is fantastic. How can I know if the two input datasets are linearly separable, so that I can use the @Amir method ?
Answers.UserDisplayName: AleWolf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T08:15:13.497Z
Answers.Text: @AleWolf I'll just put some short points.
  
  1. Visualize Data to see that the features which you've combined is linearly separable or not.
  2.  In case you are not sure or don't have any idea, there's quick and dirty way is to use *Deep Networks* instead of Neural Networks with number of layers depending on intricacy of Features. **Beware - Deep Nets require usually 2-3x more datasets than Probability based classifiers** (Usually, but that depends).
  3. The kind of function you need depends on features just like $\sin x , e^x$ are different. Prefer going to links, I've attached to see more.
Answers.UserDisplayName: Abhas Kumar Sinha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T09:13:22.250Z
Answers.Text: Thats great, last question. Why intuitively, if the two datasets are linearly separable I can use the combination of vectors method (exposed by @Amir), yet if they are not it's better to use the basis expansion methods ?
Answers.UserDisplayName: AleWolf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T12:53:14.837Z
Answers.Text: @AleWolf Linearly separable means, a straight line can easily divide the classes into 2 equal parts (which in practice, happens less often), Basis expansion gives you method to draw whatever shape of hyperplane, any curve, or even closed shapes when fitted properly (not over fitted/under fitted), they tend to separate any features as you'd like.
Answers.UserDisplayName: Abhas Kumar Sinha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T12:55:38.340Z
Answers.Text: But there are drawbacks to it (like you'd need to to have functions that are capable of that kind of curve), but when you use DeepNets, it eases the pain, since, it automatically converges to the required curve without any problem. It works by the principle of approximation and can approximate anything. Hope that answers your question... Please don't forget to read this https://towardsdatascience.com/non-linear-regression-basis-expansion-polynomials-splines-2d7adb2cc226
Answers.UserDisplayName: Abhas Kumar Sinha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:01:15.067Z
Answers.Text: What i dont understand is the relation between "concatenating two feature vectors" and their "linear separability"
Answers.UserDisplayName: AleWolf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:12:46.347Z
Answers.Text: I think you have not properly understood my original question. I'm sorry :( I've probably explained it badly. I'll try to explain it better: I've two datasets A and B. Now, **a** is a vector that belong to A and **b** is a vector that belong to B; is correct to use **a|b** as an input to a deep neural network or there are better ways to do it?
Answers.UserDisplayName: AleWolf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:23:57.730Z
Answers.Text: @AleWolf Yes, if you are using deep nets, then you can do that
Answers.UserDisplayName: Abhas Kumar Sinha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:28:01.867Z
Answers.Text: Thanks so much. Is a problem if the dimension of the combined vector is very high ? Like 16000 ? (Each feature vector has a dimension of about 8000). If yes, is there a way to deal with it ?
Answers.UserDisplayName: AleWolf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:39:03.973Z
Answers.Text: @AleWolf I'd suggest using Deep Convolution Neural Networks.. or Deep CNN
Answers.UserDisplayName: Abhas Kumar Sinha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:49:17.620Z
Answers.Text: My idea was **Radial Basis Function neural network** because the input dataset (created by concatenating the feature vectors) is quite small. What do you think ?
Answers.UserDisplayName: AleWolf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:53:33.523Z
Answers.Text: @AleWolf Sorry, I know heart of RBF ANN, but haven't experimented enough with it to suggest you that. In case you've tried that RBF/ANN before, then try experimenting/tweaking it a bit.
Answers.UserDisplayName: Abhas Kumar Sinha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:55:00.127Z
Answers.Text: So in your opinion i should try both RBF and Deep CNN and watch which performs better ?
Answers.UserDisplayName: AleWolf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:57:53.950Z
Answers.Text: @AleWolf Yep, try :)
Answers.UserDisplayName: Abhas Kumar Sinha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:58:33.490Z
Answers.Text: I really appreciate your help. Thank you infinitely :)
Answers.UserDisplayName: AleWolf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:59:38.347Z
Answers.Text: :) Keep Going..
Answers.UserDisplayName: Abhas Kumar Sinha
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T10:13:02.283Z
Answers.LastActivityDate: 2020-05-06T10:13:02.283Z
Answers.UserDisplayName: Abhas Kumar Sinha
Answers.Body: <p>The easiest way can be the concatenation of the feature vectors to create a single feature vector for each sample.</p>
  
  <p>Assume the first sample is made of the pair <span class="math-container">$X_1$</span> and <span class="math-container">$Y_1$</span>. Let the corresponding feature vectors for <span class="math-container">$X_1$</span> and <span class="math-container">$Y_1$</span> be <span class="math-container">$\textbf{v}_1$</span> and <span class="math-container">$\textbf{v}_2$</span>, respectively.</p>
  
  <p><span class="math-container">$$
  \textbf{v}_1 = [f_1, f_2, \ldots , f_n],\\
  \textbf{v}_2 = [g_1, g_2, \ldots , g_m].
  $$</span>
  Then, the first sample's feature can be defined as
  <span class="math-container">$$
  \textbf{v} = [f_1, f_2, \ldots , f_n, g_1, g_2, \ldots , g_m].
  $$</span>
  Eventually, when you pass the latter feature vector to a machine learning model, it will try to capture the dependencies among all of these features, to learn a solution for your task of interest (i.e. classification).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:15:40.340Z
Answers.Text: Is it a problem if the dimension of v1 and v2 is quite big ? Like 8000 ?
Answers.UserDisplayName: AleWolf
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-10T17:27:45.930Z
Answers.Text: It also depends on the number of samples that you have in your dataset. In other words, you need enough samples to be able to solve your problem in that high dimensional space. If you want to learn more please refer to subsection 5.11.1 The curse of dimensionality on the Deep Learning book (available at https://www.deeplearningbook.org/) [1].
  As a quick solution, you may use dimensionality reduction techniques to reduce the number of features that you have.
  [1] Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Answers.UserDisplayName: Amir
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-06T00:27:33.683Z
Answers.LastActivityDate: 2020-05-06T00:27:33.683Z
Answers.UserDisplayName: Amir
Body: <p>Let's consider this scenario. I have two conceptually different video datasets, for example a dataset A composed of videos about cats and a dataset B composed of videos about houses. Now, <strong>I'm able</strong> to extract a feature vectors from both the samples of the datasets A and B, and I know that, each sample in the dataset A is related to one and only one sample in the dataset B and they belong to a specific class (there are only 2 classes).</p>
  
  <p>For example:</p>
  
  <pre><code>Sample x1 AND sample y1 ---&gt; Class 1
  Sample x2 AND sample y2 ---&gt; Class 2
  Sample x3 AND sample y3 ---&gt; Class 1
  and so on...
  </code></pre>
  
  <p><strong>If I extract the feature vectors from samples in both datasets , which is the best way to combine them in order to give a correct input to the classifier (for example a neural network) ?</strong></p>
  
  <p>feature vector <strong>v1</strong> extracted from <strong>x1</strong> + feature vector <strong>v1'</strong> extracted from <strong>y1</strong> ---> input for classifier</p>
  
  <p><em>I ask this because I suspect that neural networks only take one vector as input, while I have to combine two vectors</em></p>
  
Tags: <neural-networks><deep-learning><classification>
Title: Combine two feature vectors for a correct input of a neural network
UserDisplayName: AleWolf
ThreadText: # Combine two feature vectors for a correct input of a neural network
  
  (Asked by: username_0 on 2020-05-05)
  
  Let's consider this scenario. I have two conceptually different video datasets, for example a dataset A composed of videos about cats and a dataset B composed of videos about houses. Now, I'm able to extract a feature vectors from both the samples of the datasets A and B, and I know that, each sample in the dataset A is related to one and only one sample in the dataset B and they belong to a specific class (there are only 2 classes).
  For example:
  Sample x1 AND sample y1 ---> Class 1
  Sample x2 AND sample y2 ---> Class 2
  Sample x3 AND sample y3 ---> Class 1
  and so on...
  
  If I extract the feature vectors from samples in both datasets , which is the best way to combine them in order to give a correct input to the classifier (for example a neural network) ?
  feature vector v1 extracted from x1 + feature vector v1' extracted from y1 ---> input for classifier
  I ask this because I suspect that neural networks only take one vector as input, while I have to combine two vectors
  
  **Tags:** neural-networks, deep-learning, classification
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-05-06. Score: 3 | Accepted answer
  
  The easiest way can be the concatenation of the feature vectors to create a single feature vector for each sample.
  
  Assume the first sample is made of the pair $X\_1$ and $Y\_1$. Let the corresponding feature vectors for $X\_1$ and $Y\_1$ be $\textbf{v}\_1$ and $\textbf{v}\_2$, respectively.
  
  $$
  \textbf{v}\_1 = [f\_1, f\_2, \ldots , f\_n],\\
  \textbf{v}\_2 = [g\_1, g\_2, \ldots , g\_m].
  $$
  Then, the first sample's feature can be defined as
  $$
  \textbf{v} = [f\_1, f\_2, \ldots , f\_n, g\_1, g\_2, \ldots , g\_m].
  $$
  Eventually, when you pass the latter feature vector to a machine learning model, it will try to capture the dependencies among all of these features, to learn a solution for your task of interest (i.e. classification).
  
  > **Comments:**
  > **username_0**: Is it a problem if the dimension of v1 and v2 is quite big ? Like 8000 ?  
  > **username_2**: It also depends on the number of samples that you have in your dataset. In other words, you need enough samples to be able to solve your problem in that high dimensional space. If you want to learn more please refer to subsection 5.11.1 The curse of dimensionality on the Deep Learning book (available at https://www.deeplearningbook.org/) [1]. As a quick solution, you may use dimensionality reduction techniques to reduce the number of features that you have. [1] Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.  
  
  ---
  
  ### Answer by username_1 on 2020-05-06. Score: 1
  
  *$^\*$Note - Question is bit unclear, in case the answer doesn't addresses the question, please ask for edit/delete Request.*
  
  ---
  
  ## GENERALIZATION
  
  Suppose there are multiple datasets denoted by $A\_i$. Datasets contain a set of Vectors $x\_{j} $. Mathematically $A\_i = \{ x\_j\}\_{j=0}^n$. We've to find an estimator function $\hat f$, such that $\hat f( \vec r) = y, \, \vec r \in X$ where $X $ is a special dataset created by combining all $A\_i$ which helps in classification into $y \in Y$ which is the set of classes.
  
  .
  
  As @Amir Mentioned out, linearly separable feature can be easily separated by straight combination of vectors i.e. if $x\_u \in A\_i, w\_v \in A\_j \dots$, then $r = [x\_1 \,x\_2 \, \dots \, x\_u \, w\_1 \, \dots w\_v \dots]$. Where, $r \in X$ which is the required dataset.
  
  *There are cases where the features are not linearly separable*, We use **basis expansion methods**[[1]](https://towardsdatascience.com/non-linear-regression-basis-expansion-polynomials-splines-2d7adb2cc226) to make required shape of hyperplane to separate the features. We create a new dataset combining $A\_i \, \forall i \in C \subset \mathbb N$. Suppose that the new dataset is $X$, then $r \in X$ and $r = [r\_0, r\_1, \dots r\_n].$
  
  Then,
  
  $$r\_1 = u\_1^2v\_1^2 \\ r\_2 = \sin(u\_2)\sin(v\_2) \\ r\_3 = ae^{u\_3 + v\_3} \\ r\_4 = a v\_4 v\_4 + a\_2 u\_4^2 v\_4^2 + \dots \\ \dots$$
  
  Here $u\_p \in A\_i; \, v\_q \in A\_j$
  
  Here you can use all the creativity to set $r = [r\_1, r\_2, \dots , r\_n]$ and make a new dataset. What equations and what functions you chose fully depends on the kind of hyperplane shape you want to obtain. Basis expansion is just one of the methods for feature extraction is certainly one of the most flexible too.
  
  Now, you feed the newly created vectors into your trained estimator functions (which is Neural Net) which can classify things much easily now.
  
  *In case of Regression/Classification without Neural Net needs some extra treatment to train the model*[[2]](https://medium.com/datadriveninvestor/improve-your-classification-models-using-mean-target-encoding-a3d573df31e8).
  
  ---
  
  **[[2]](https://medium.com/datadriveninvestor/improve-your-classification-models-using-mean-target-encoding-a3d573df31e8)Note:** There is also a big role of encoding. For example, if you encode colors by numbers $1, 2, 3$ for RGB or $10,01, 11$ fully changes everything and your features too. In such cases, You may even need different equations to make your required dataset $X$ and vectors $r$.
  
  ---
  
  ## REFERENCES:
  
  1. Oleszak, Michal. <https://towardsdatascience.com>. [*Non-linear regression: basis expansion, polynomials & splines*](https://towardsdatascience.com/non-linear-regression-basis-expansion-polynomials-splines-2d7adb2cc226). Sep 30, 2019. Web. 6 May 2020.
  2. Sangarshanan. <https://medium.com>. [*Improve your classification models using Mean /Target Encoding*](https://medium.com/datadriveninvestor/improve-your-classification-models-using-mean-target-encoding-a3d573df31e8). Jun 23, 2018. Web. 6 May 2020.
  
  > **Comments:**
  > **username_0**: I really appreciate your help, your answer is fantastic. How can I know if the two input datasets are linearly separable, so that I can use the @Amir method ?  
  > **username_1**: @AleWolf I'll just put some short points.  1. Visualize Data to see that the features which you've combined is linearly separable or not. 2.  In case you are not sure or don't have any idea, there's quick and dirty way is to use *Deep Networks* instead of Neural Networks with number of layers depending on intricacy of Features. **Beware - Deep Nets require usually 2-3x more datasets than Probability based classifiers** (Usually, but that depends). 3. The kind of function you need depends on features just like $\sin x , e^x$ are different. Prefer going to links, I've attached to see more.  
  > **username_0**: Thats great, last question. Why intuitively, if the two datasets are linearly separable I can use the combination of vectors method (exposed by @Amir), yet if they are not it's better to use the basis expansion methods ?  
  > **username_1**: @AleWolf Linearly separable means, a straight line can easily divide the classes into 2 equal parts (which in practice, happens less often), Basis expansion gives you method to draw whatever shape of hyperplane, any curve, or even closed shapes when fitted properly (not over fitted/under fitted), they tend to separate any features as you'd like.  
  > **username_1**: But there are drawbacks to it (like you'd need to to have functions that are capable of that kind of curve), but when you use DeepNets, it eases the pain, since, it automatically converges to the required curve without any problem. It works by the principle of approximation and can approximate anything. Hope that answers your question... Please don't forget to read this https://towardsdatascience.com/non-linear-regression-basis-expansion-polynomials-splines-2d7adb2cc226  
  > **username_0**: What i dont understand is the relation between "concatenating two feature vectors" and their "linear separability"  
  > **username_0**: I think you have not properly understood my original question. I'm sorry :( I've probably explained it badly. I'll try to explain it better: I've two datasets A and B. Now, **a** is a vector that belong to A and **b** is a vector that belong to B; is correct to use **a|b** as an input to a deep neural network or there are better ways to do it?  
  > **username_1**: @AleWolf Yes, if you are using deep nets, then you can do that  
  > **username_0**: Thanks so much. Is a problem if the dimension of the combined vector is very high ? Like 16000 ? (Each feature vector has a dimension of about 8000). If yes, is there a way to deal with it ?  
  > **username_1**: @AleWolf I'd suggest using Deep Convolution Neural Networks.. or Deep CNN  
  > **username_0**: My idea was **Radial Basis Function neural network** because the input dataset (created by concatenating the feature vectors) is quite small. What do you think ?  
  > **username_1**: @AleWolf Sorry, I know heart of RBF ANN, but haven't experimented enough with it to suggest you that. In case you've tried that RBF/ANN before, then try experimenting/tweaking it a bit.  
  > **username_0**: So in your opinion i should try both RBF and Deep CNN and watch which performs better ?  
  > **username_1**: @AleWolf Yep, try :)  
  > **username_0**: I really appreciate your help. Thank you infinitely :)  
  > **username_1**: :) Keep Going..  
  
  ---
  
  