Answers.Body: <p>Feature engineering may be <em>necessary</em> when one cannot achieve acceptable error rate — within a budget or in principle.</p>
  <p>NN may be stalling due to information bottleneck: too many pigeons, not enough holes. In that case, custom features may provide slightly better information compression. (Alas, this is not a panacea: some layer(s) may still be too narrow. That's why starting fat and pruning later deems superior, although not always affordable.)</p>
  <p>Not a necessity, but a no-brainer nonetheless: having strong insight into underlying processes is a clear call for feature engineering. Let's say, it's apparent that known meaningful transformation can hardly be approximated by (reasonably small) subnetwork: mapping between periodic and non-periodic functions, as an example.</p>
  <p>Personally in my check-list, feature stage is coupled with a reminder to search for a domain expert. In some practical sense features are being extracted from people rather than data. Our neural networks are pretty good too!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-16T11:45:47.360Z
Answers.LastActivityDate: 2021-03-16T11:45:47.360Z
Answers.UserDisplayName: ShadowsInRain
Answers.Body: <ul>
  <li>normalization &amp; scaling  increase speed of learning...</li>
  <li>first &amp; second differencies, that mentioned <em>@Vladislav Gladkikh</em>, are usefull when autocorrelation in series exists (either timeseries or the other) -- any correlations can be seen in EDA-stage of analysis, though be carefull about some traps in correlations...</li>
  <li>for sparse data - use sparse-to-dense technique for speed as well (I think, NN copes with it with adequate architecture of it), in optimization algorithms thi stage is known as factorization or preconditioning stage</li>
  <li>or dense-to-sparse as regularization stage or for whitening (decorrelating) your data (as autoencoder)</li>
  <li>for categorical features encoding is obligatory</li>
  </ul>
  <p>In any case, Factor Analysis (Exploratory Data Analysis &amp; Confirmatory Data Analysis) is preferable before constructing the model either with Machine Learning or Deep Learning</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-02T07:42:17.510Z
Answers.LastActivityDate: 2024-06-02T07:42:17.510Z
Answers.UserDisplayName: JeeyCi
Answers.Body: <p>Yes, neural networks learn features themselves freeing you from the need to manually engineer them. I will illustrate it here with a toy problem.</p>
  <p>Let's assume that we want to learn the areas of parallelograms built on pairs of vectors:<a href="https://i.sstatic.net/hwKdvm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hwKdvm.png" alt="enter image description here" /></a></p>
  <p>The input data are six coordinates: <span class="math-container">$(x_1, y_1, x_2, y_2, x_3, y_3)$</span>.</p>
  <pre><code>import numpy as np
  
  n_tr = 1000 # training data
  x_tr = np.random.uniform(low=-1.0, high=1.0, size=(n_tr, 6))
  
  n_ts = 100 # test data                                        
  x_ts = np.random.uniform(low=-1.0, high=1.0, size=(n_ts, 6)) 
  </code></pre>
  <p>The targets (areas) are <span class="math-container">$y = |ad-bc|$</span>, where <span class="math-container">$a=x_3-x_1$</span>, <span class="math-container">$b=y_3-y_1$</span>, <span class="math-container">$c=x_2-x_1$</span>, <span class="math-container">$d=y_2-y_1$</span>.</p>
  <pre><code>a_tr = x_tr[:,4] - x_tr[:,0] # x_3 - x_1
  b_tr = x_tr[:,5] - x_tr[:,1] # y_3 - y_1
  c_tr = x_tr[:,2] - x_tr[:,0] # x_2 - x_1
  d_tr = x_tr[:,3] - x_tr[:,1] # y_2 - y_1
  y_tr = np.abs(a_tr*d_tr-b_tr*c_tr)
  
  a_ts = x_ts[:,4] - x_ts[:,0] # x_3 - x_1
  b_ts = x_ts[:,5] - x_ts[:,1] # y_3 - y_1
  c_ts = x_ts[:,2] - x_ts[:,0] # x_2 - x_1
  d_ts = x_ts[:,3] - x_ts[:,1] # y_2 - y_1
  y_ts = np.abs(a_ts*d_ts-b_ts*c_ts)
  </code></pre>
  <p>To learn the areas from coordinates, I will use my favorite machine learning library <code>super_magic_learn</code></p>
  <pre><code>from super_magic_learn import wonder_network
  
  wonder_network.init()
  </code></pre>
  <p>It will initialize a network with random activation functions in neurons, and random connections between them having random weights. It also randomly assigns some neurons as inputs, while other outputs or internal ones.</p>
  <p><a href="https://i.sstatic.net/9vNYXm.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9vNYXm.png" alt="enter image description here" /></a></p>
  <p>Then I train my network</p>
  <pre><code>wonder_network.fit(x_tr, y_tr, use_wand=True)
  </code></pre>
  <p>During training, the activation functions inside neurons change, the connections between neurons form, disappear, and form again, and their weights are adjusted. Some neurons organize in layers, the number of neurons in each layer changes, and finally the trained network is as follows:</p>
  <p><a href="https://i.sstatic.net/fiRRCl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fiRRCl.png" alt="enter image description here" /></a></p>
  <p>It solves the task with 100% accuracy for both the training and test data, and it solves it using only raw data: coordinates. No need to engineer features.</p>
  <p>However, you probably don't have access to the library <code>super_magic_learn</code>. Let's see what can we do with a slightly more inferior <code>tensorflow</code></p>
  <pre><code>import tensorflow as tf
  from tensorflow import keras
  from tensorflow.keras import layers
  
  model = keras.Sequential(
      [
          layers.Dense(64, activation=&quot;tanh&quot;, input_dim=6),
          layers.Dense(4, activation=&quot;tanh&quot;),
          layers.Dense(1),
      ]
  )
  
  optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)
  
  model.compile(loss='mean_squared_error',
                  optimizer=optimizer,
                  metrics=['mean_squared_error'])
  
  model.fit(x_tr, y_tr, epochs=512, batch_size=64, validation_split = 0.2, verbose=1)
  y_pr = model.predict(x_ts)
  </code></pre>
  <p>Now calculate the performance on the test set</p>
  <pre><code>from sklearn.metrics import r2_score
  print('Rsq:', r2_score(y_ts,y_pr))
  </code></pre>
  <p><span class="math-container">$R^2$</span>:</p>
  <pre><code>Rsq: 0.4802872880495598
  </code></pre>
  <p>Not good. What will happen, if I engineer some features?</p>
  <p>Let's train the same model but instead of feeding it with raw data, the inputs will be the following manually engineered features: <span class="math-container">$a=x_3-x_1$</span>, <span class="math-container">$b=y_3-y_1$</span>, <span class="math-container">$c=x_2-x_1$</span>, <span class="math-container">$d=y_2-y_1$</span> (don't forget to change <code>input_dim=4</code> in the first layer).</p>
  <pre><code>x_tr = np.c_[a_tr, b_tr, c_tr, d_tr]
  x_ts = np.c_[a_ts, b_ts, c_ts, d_ts]
  </code></pre>
  <p><span class="math-container">$R^2$</span>:</p>
  <pre><code>Rsq: 0.8841499533897564
  </code></pre>
  <p>Now it is much better. Less than 100% though.</p>
  <p>Why neural network in <code>tensorflow</code> performs poorly on raw data and needs feature engineering while the <code>super_magic_learn</code> works perfectly on raw data and does not need any feature engineering?</p>
  <p>The reason is that <code>tensorflow</code> or any other library that I know, is much more restricted than my beloved <code>super_magic_learn</code>. The restrictions are as follows (note a very small problem though: <code>super_magic_learn</code> does not exist but I wish it were):</p>
  <ul>
  <li>You have only a very small number of activation functions to choose from, like tanh, relu and a handful of others.</li>
  <li>The activation functions stay fixed during training. You cannot change them.</li>
  <li>You cannot add/remove layers.</li>
  <li>You cannot change the number of neurons in the layers.</li>
  <li>You cannot add/remove connections between the neurons.</li>
  <li>You have to organize your neurons only in layers, no other arrangement is allowed.</li>
  <li>During training, the network cannot learn the most suitable architecture for the task. E.g., it cannot reorganize itself taking into account the symmetries of the problem.</li>
  <li>etc ...</li>
  <li>Basically, the only thing you can do during training is to learn the weights.</li>
  </ul>
  <p>The textbooks are right: ideally, a neural network should learn just from the raw data. But this is true only about my idealized library and not so much about existing real-world implementations.</p>
  <p>To make a network really learn features for any task, it should be freed from these restrictions.</p>
  <p>If you put so many restrictions on the architecture, activation functions, and other parameters, so that they cannot be learned from the data during training, then you have to engineer them yourself and adjust them manually for your task. If you engineer them correctly then your network will learn happily from the raw data. But it might perform poorly on other tasks.</p>
  <p>Such is the case with convolutional neural networks. They were designed taking into account transnational equivariance of features in images that's why they can learn features from raw image data. However, they don't necessarily perform well in other domains.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-17T15:22:09.473Z
Answers.Text: Great post, though I have a question about magic learn approach -- wouldn't described newtork overfit and can it approximate functions like sine non-locally?
Answers.UserDisplayName: Kirill Fedyanin
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T01:22:08.307Z
Answers.Text: @KirillFedyanin Easy! It detects symmetries which is demonstrated in my example where it sensed that the answer is translation invariant and depends only on the differences between the corresponding coordinates. In the case of sine, it will detect periodicity and solve it by choosing periodic activations. It also chooses multiplicative aggregation of inputs where appropriate which helps a lot. As to overfitting, the `fit` method has an option `buben_dance=True` (this is default) - it detects the right complexity at each feature level and adjusts the number of neurons in each layer accordingly.
Answers.UserDisplayName: Vladislav Gladkikh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T05:58:37.603Z
Answers.Text: This kind of "magic ML" is called neuro-evolution, but it's non used often because it **does not scale** well. Mainstream ML architectures are wired in static way and in layers because ML hardware (mostly GPUs) is very good at handling long sequences of data in parallel, but will choke on logic. That's why Tensorflow and friends have all these limitations.
Answers.UserDisplayName: ShadowsInRain
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-25T13:12:56.680Z
Answers.Text: Thanks for answer. I really appreciated & satisfied.
Answers.UserDisplayName: dasmehdix
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-17T12:46:21.897Z
Answers.LastActivityDate: 2021-03-17T12:46:21.897Z
Answers.UserDisplayName: Vladislav Gladkikh
Body: <p>Usually, Neural Networks uses raw data. You do not need to extract features manually. NN's can find &amp; extract good features which is a pattern of an image, signal or any kind of data. When we check layer outputs in a NN, we can see and visualize how NNs extract features.</p>
  <p>Do neural networks extract features by themselves every time? When is it necessary to manually extract or engineer features to feed into the neural network rather than providing raw data?</p>
  <p>For example, I had a time series sensor data. When I use LSTM &amp; GRU on a raw dataset, I had bad test accuracy but when I extract some features manually I had really good test set accuracy results. I extract Fast Fourier Transform, Cross-correlation features which helped a lot to increase accuracy. &quot;Extraction of features manually&quot; helped to solve my problem.</p>
  
Tags: <neural-networks><deep-learning><feature-extraction><features><feature-engineering>
Title: When is it necessary to manually extract features to feed into the neural network rather than providing raw data?
UserDisplayName: dasmehdix
ThreadText: # When is it necessary to manually extract features to feed into the neural network rather than providing raw data?
  
  (Asked by: username_0 on 2021-03-08)
  
  Usually, Neural Networks uses raw data. You do not need to extract features manually. NN's can find & extract good features which is a pattern of an image, signal or any kind of data. When we check layer outputs in a NN, we can see and visualize how NNs extract features.
  
  Do neural networks extract features by themselves every time? When is it necessary to manually extract or engineer features to feed into the neural network rather than providing raw data?
  
  For example, I had a time series sensor data. When I use LSTM & GRU on a raw dataset, I had bad test accuracy but when I extract some features manually I had really good test set accuracy results. I extract Fast Fourier Transform, Cross-correlation features which helped a lot to increase accuracy. "Extraction of features manually" helped to solve my problem.
  
  **Tags:** neural-networks, deep-learning, feature-extraction, features, feature-engineering
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2021-03-17. Score: 3 | Accepted answer
  
  Yes, neural networks learn features themselves freeing you from the need to manually engineer them. I will illustrate it here with a toy problem.
  
  Let's assume that we want to learn the areas of parallelograms built on pairs of vectors:[![enter image description here](https://i.sstatic.net/hwKdvm.png)](https://i.sstatic.net/hwKdvm.png)
  
  The input data are six coordinates: $(x\_1, y\_1, x\_2, y\_2, x\_3, y\_3)$.
  
  ```
  import numpy as np
  
  n_tr = 1000 # training data
  x_tr = np.random.uniform(low=-1.0, high=1.0, size=(n_tr, 6))
  
  n_ts = 100 # test data                                        
  x_ts = np.random.uniform(low=-1.0, high=1.0, size=(n_ts, 6)) 
  
  ```
  
  The targets (areas) are $y = |ad-bc|$, where $a=x\_3-x\_1$, $b=y\_3-y\_1$, $c=x\_2-x\_1$, $d=y\_2-y\_1$.
  
  ```
  a_tr = x_tr[:,4] - x_tr[:,0] # x_3 - x_1
  b_tr = x_tr[:,5] - x_tr[:,1] # y_3 - y_1
  c_tr = x_tr[:,2] - x_tr[:,0] # x_2 - x_1
  d_tr = x_tr[:,3] - x_tr[:,1] # y_2 - y_1
  y_tr = np.abs(a_tr*d_tr-b_tr*c_tr)
  
  a_ts = x_ts[:,4] - x_ts[:,0] # x_3 - x_1
  b_ts = x_ts[:,5] - x_ts[:,1] # y_3 - y_1
  c_ts = x_ts[:,2] - x_ts[:,0] # x_2 - x_1
  d_ts = x_ts[:,3] - x_ts[:,1] # y_2 - y_1
  y_ts = np.abs(a_ts*d_ts-b_ts*c_ts)
  
  ```
  
  To learn the areas from coordinates, I will use my favorite machine learning library `super_magic_learn`
  
  ```
  from super_magic_learn import wonder_network
  
  wonder_network.init()
  
  ```
  
  It will initialize a network with random activation functions in neurons, and random connections between them having random weights. It also randomly assigns some neurons as inputs, while other outputs or internal ones.
  
  [![enter image description here](https://i.sstatic.net/9vNYXm.png)](https://i.sstatic.net/9vNYXm.png)
  
  Then I train my network
  
  ```
  wonder_network.fit(x_tr, y_tr, use_wand=True)
  
  ```
  
  During training, the activation functions inside neurons change, the connections between neurons form, disappear, and form again, and their weights are adjusted. Some neurons organize in layers, the number of neurons in each layer changes, and finally the trained network is as follows:
  
  [![enter image description here](https://i.sstatic.net/fiRRCl.png)](https://i.sstatic.net/fiRRCl.png)
  
  It solves the task with 100% accuracy for both the training and test data, and it solves it using only raw data: coordinates. No need to engineer features.
  
  However, you probably don't have access to the library `super_magic_learn`. Let's see what can we do with a slightly more inferior `tensorflow`
  
  ```
  import tensorflow as tf
  from tensorflow import keras
  from tensorflow.keras import layers
  
  model = keras.Sequential(
      [
          layers.Dense(64, activation="tanh", input_dim=6),
          layers.Dense(4, activation="tanh"),
          layers.Dense(1),
      ]
  )
  
  optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)
  
  model.compile(loss='mean_squared_error',
                  optimizer=optimizer,
                  metrics=['mean_squared_error'])
  
  model.fit(x_tr, y_tr, epochs=512, batch_size=64, validation_split = 0.2, verbose=1)
  y_pr = model.predict(x_ts)
  
  ```
  
  Now calculate the performance on the test set
  
  ```
  from sklearn.metrics import r2_score
  print('Rsq:', r2_score(y_ts,y_pr))
  
  ```
  
  $R^2$:
  
  ```
  Rsq: 0.4802872880495598
  
  ```
  
  Not good. What will happen, if I engineer some features?
  
  Let's train the same model but instead of feeding it with raw data, the inputs will be the following manually engineered features: $a=x\_3-x\_1$, $b=y\_3-y\_1$, $c=x\_2-x\_1$, $d=y\_2-y\_1$ (don't forget to change `input_dim=4` in the first layer).
  
  ```
  x_tr = np.c_[a_tr, b_tr, c_tr, d_tr]
  x_ts = np.c_[a_ts, b_ts, c_ts, d_ts]
  
  ```
  
  $R^2$:
  
  ```
  Rsq: 0.8841499533897564
  
  ```
  
  Now it is much better. Less than 100% though.
  
  Why neural network in `tensorflow` performs poorly on raw data and needs feature engineering while the `super_magic_learn` works perfectly on raw data and does not need any feature engineering?
  
  The reason is that `tensorflow` or any other library that I know, is much more restricted than my beloved `super_magic_learn`. The restrictions are as follows (note a very small problem though: `super_magic_learn` does not exist but I wish it were):
  
  * You have only a very small number of activation functions to choose from, like tanh, relu and a handful of others.
  * The activation functions stay fixed during training. You cannot change them.
  * You cannot add/remove layers.
  * You cannot change the number of neurons in the layers.
  * You cannot add/remove connections between the neurons.
  * You have to organize your neurons only in layers, no other arrangement is allowed.
  * During training, the network cannot learn the most suitable architecture for the task. E.g., it cannot reorganize itself taking into account the symmetries of the problem.
  * etc ...
  * Basically, the only thing you can do during training is to learn the weights.
  
  The textbooks are right: ideally, a neural network should learn just from the raw data. But this is true only about my idealized library and not so much about existing real-world implementations.
  
  To make a network really learn features for any task, it should be freed from these restrictions.
  
  If you put so many restrictions on the architecture, activation functions, and other parameters, so that they cannot be learned from the data during training, then you have to engineer them yourself and adjust them manually for your task. If you engineer them correctly then your network will learn happily from the raw data. But it might perform poorly on other tasks.
  
  Such is the case with convolutional neural networks. They were designed taking into account transnational equivariance of features in images that's why they can learn features from raw image data. However, they don't necessarily perform well in other domains.
  
  > **Comments:**
  > **username_4**: Great post, though I have a question about magic learn approach -- wouldn't described newtork overfit and can it approximate functions like sine non-locally?  
  > **username_3**: @username_4 Easy! It detects symmetries which is demonstrated in my example where it sensed that the answer is translation invariant and depends only on the differences between the corresponding coordinates. In the case of sine, it will detect periodicity and solve it by choosing periodic activations. It also chooses multiplicative aggregation of inputs where appropriate which helps a lot. As to overfitting, the `fit` method has an option `buben_dance=True` (this is default) - it detects the right complexity at each feature level and adjusts the number of neurons in each layer accordingly.  
  > **username_1**: This kind of "magic ML" is called neuro-evolution, but it's non used often because it **does not scale** well. Mainstream ML architectures are wired in static way and in layers because ML hardware (mostly GPUs) is very good at handling long sequences of data in parallel, but will choke on logic. That's why Tensorflow and friends have all these limitations.  
  > **username_0**: Thanks for answer. I really appreciated & satisfied.  
  
  ---
  
  ### Answer by username_1 on 2021-03-16. Score: 2
  
  Feature engineering may be *necessary* when one cannot achieve acceptable error rate — within a budget or in principle.
  
  NN may be stalling due to information bottleneck: too many pigeons, not enough holes. In that case, custom features may provide slightly better information compression. (Alas, this is not a panacea: some layer(s) may still be too narrow. That's why starting fat and pruning later deems superior, although not always affordable.)
  
  Not a necessity, but a no-brainer nonetheless: having strong insight into underlying processes is a clear call for feature engineering. Let's say, it's apparent that known meaningful transformation can hardly be approximated by (reasonably small) subnetwork: mapping between periodic and non-periodic functions, as an example.
  
  Personally in my check-list, feature stage is coupled with a reminder to search for a domain expert. In some practical sense features are being extracted from people rather than data. Our neural networks are pretty good too!
  
  
  ---
  
  ### Answer by username_2 on 2024-06-02. Score: 0
  
  * normalization & scaling increase speed of learning...
  * first & second differencies, that mentioned *@username_3*, are usefull when autocorrelation in series exists (either timeseries or the other) -- any correlations can be seen in EDA-stage of analysis, though be carefull about some traps in correlations...
  * for sparse data - use sparse-to-dense technique for speed as well (I think, NN copes with it with adequate architecture of it), in optimization algorithms thi stage is known as factorization or preconditioning stage
  * or dense-to-sparse as regularization stage or for whitening (decorrelating) your data (as autoencoder)
  * for categorical features encoding is obligatory
  
  In any case, Factor Analysis (Exploratory Data Analysis & Confirmatory Data Analysis) is preferable before constructing the model either with Machine Learning or Deep Learning
  
  
  ---
  
  