Answers.Body: <p>I have very little experience with ML/DL to call myself either practitioner, but here is my answer on the 1st question:</p>
  
  <p>At its core DL solves well the task of classification. Not every practical problem can be rephrased in terms of classification. Classification domain needs to be known upfront. Although the classification can be applied to any type of data, it's necessary to train the NN with samples of the specific domain where it'll be applied. If the domain is switched at some point, while keeping the same model (NN structure), it'll have to be retrained with new samples. Furthermore, even the best classifiers have "gaps" - <a href="http://www.kdnuggets.com/2015/07/deep-learning-adversarial-examples-misconceptions.html" rel="noreferrer">Adversarial Examples</a> can be easily constructed from a training sample, such that changes are imperceptible to human, but are misclassified by the trained model.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-12-14T21:41:20.370Z
Answers.Text: 'Classification' can be considered a special case of 'regression', which is probably therefore a better characterization of DL.
Answers.UserDisplayName: NietzscheanAI
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-13T22:20:08.543Z
Answers.LastActivityDate: 2016-10-13T22:20:08.543Z
Answers.UserDisplayName: Iliyan Bobev
Answers.Body: <p>Question 2.
  I am researching whether Hyper dimensional computing is an alternative to Deep Learning.  Hyper-D uses very long bit vectors (10,000 bits) to encode information.  The vectors are random and as such they are approximately orthogonal.  By grouping and averaging a collection of such vectors a "set" can be formed and later queried to see if an unknown vector belongs to the set.  The set can be considered a concept or a generalize image, etc.  Training is very fast as is recognition.  What needs to be done is simulate the domains in which Deep Learning has been successful and compare Hyper-D with it.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-26T17:58:39.940Z
Answers.Text: Interesting. So how does this differ from Kanerva's 'Sparse Distributed Memory'?
Answers.UserDisplayName: NietzscheanAI
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-28T04:09:04.220Z
Answers.Text: Both are developed by Pentti Kanerva.  Look up Hyper dimensional computing to see the difference.  Too long to answer here.
Answers.UserDisplayName: Douglas G Danforth
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-04-04T08:09:51.477Z
Answers.LastActivityDate: 2017-04-04T08:09:51.477Z
Answers.UserDisplayName: Douglas G Danforth
Answers.Body: <p>From a mathematics point of view one of the major issues in deep networks with several layers are <strong>vanishing</strong> or <strong>unstable gradients</strong>. Each additional hidden layer learns significantly slower, almost nullifying the benefit of the additional layer.</p>
  
  <p>Modern deep learning approaches can improve this behavior, but in simple, old fashioned neural networks this is a well known issue. You can find a well written analysis <a href="http://neuralnetworksanddeeplearning.com/chap5.html" rel="nofollow noreferrer">here</a> for deeper study.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-21T21:24:12.957Z
Answers.LastActivityDate: 2017-08-21T21:24:12.957Z
Answers.UserDisplayName: Demento
Answers.Body: <p>To summarize, There are two major issues in applied Deep Learning.</p>
  
  <ul>
  <li><p>The first being that computationally , it's exhaustive. Normal CPU's require a lot of time to perform even the basic computation/training with Deep Learning. GPU's are thus recommended however, even they may not be enough in a lot of situations. Typical deep learning models don't support the theoretical time to be in Polynomials. However, if we look at the relatively simpler models in ML for the same tasks, too often we have mathematical guarantees that training time required for such simpler Algorithms is in Polynomials. This, for me, at least is probably the biggest difference. </p>
  
  <p>There're solutions to counter this issue, though. One main approach being is to optimize DL Algorithms to a number of iterations only (instead of looking at the global solutions in practice, just optimize the algorithm to a good local solution, whereas the criterion for "Good" is defined by the user).</p></li>
  <li><p>Another Issue which may be a little bit controversial to young deep learning enthusiasts is that Deep Learning algorithms lack theoretical understanding and reasoning. Deep Neural Networks have been successfully used in a lot of situations including Hand writing recognition, Image processing, Self Driving Cars, Signal Processing, NLP and Biomedical Analysis. In some of these cases, they have even surpassed humans. However, that being said, they're not under any circumstance, theoretically as sound as most of Statistical Methods.</p>
  
  <p>I will not go into detail , rather I leave that up to you. There're pros and cons for every Algorithm/methodology and DL is not an exception. It's very useful as has been proven in a lot of situations and every young Data Scientist must learn at least the basics of DL. However, in the case of relatively simple problems, it's better to use famous Statistical methods as they have a lot of theoretical results/guarantees to support them. Furthermore, from learning point of view, it's always better to start with simple approaches and master them first. </p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-26T17:56:20.817Z
Answers.Text: By 'in polynomials' you mean 'in polynomial time', right? Have you got a reference to support that?
Answers.UserDisplayName: NietzscheanAI
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-27T08:23:50.730Z
Answers.Text: Yes, that's exactly what I mean. Sure, It can be proved in a lot of situations...I will start with the Simplest possible example, Just training a Network with three Nodes, and two layers is NP-Complete problem as shown here.(http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.42.6662&rep=rep1&type=pdf) . Remember that this paper is very old, and now we have more ideas on how to improve in practice, with some heuristics, but still, theoretically, there're no improved results.
Answers.UserDisplayName: Sibghat Ullah
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-27T08:28:04.367Z
Answers.Text: Other nice article on the same issue, which also describes some tricks to improve training time in practice. (https://pdfs.semanticscholar.org/9499/140ffba43fd87d84bebc2fc328fa147ac236.pdf)
Answers.UserDisplayName: Sibghat Ullah
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-27T08:36:08.020Z
Answers.Text: Let's say, we want to predict the price for something. Simple Linear Regression with least square fit will have a Polynomial time, whereas solving the same issue with Neural Networks (even the simplest of them) will result in NP complete problem. This is a very big difference. Eventually, you have to carefully select an algorithm for a specific task. For example, Least Square fit has specific assumptions, which includes , "The ideal function which the algorithm is learning, can be learned as linear combination of features". If that assumption is not valid,  so are results achieved.
Answers.UserDisplayName: Sibghat Ullah
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-27T15:46:51.440Z
Answers.Text: Of course,simply because a problem (in this case, finding optimal weights) is NP-complete does not in itself mean that there aren't efficient practical methods for finding _good_ weights...
Answers.UserDisplayName: NietzscheanAI
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-26T13:17:52.717Z
Answers.LastActivityDate: 2018-03-26T13:17:52.717Z
Answers.UserDisplayName: Sibghat Ullah
Body: <p>Over the last 50 years, the rise/fall/rise in popularity of neural nets has acted as something of a 'barometer' for AI research.</p>
  
  <p>It's clear from the questions on this site that people are interested in applying Deep Learning (DL) to a wide variety of difficult problems.</p>
  
  <p>I therefore have two questions:</p>
  
  <ol>
  <li>Practitioners - What do you find to be the main obstacles to
  applying DL 'out of the box' to your problem? </li>
  <li>Researchers - What
  techniques do you use (or have developed) that might help address
  practical issues? Are they within DL or do they offer an
  alternative approach?</li>
  </ol>
  
Tags: <deep-learning>
Title: Issues with and alternatives to Deep Learning approaches?
UserDisplayName: NietzscheanAI
ThreadText: # Issues with and alternatives to Deep Learning approaches?
  
  (Asked by: username_0 on 2016-08-03)
  
  Over the last 50 years, the rise/fall/rise in popularity of neural nets has acted as something of a 'barometer' for AI research.
  
  It's clear from the questions on this site that people are interested in applying Deep Learning (DL) to a wide variety of difficult problems.
  
  I therefore have two questions:
  
  1. Practitioners - What do you find to be the main obstacles to
     applying DL 'out of the box' to your problem?
  2. Researchers - What
     techniques do you use (or have developed) that might help address
     practical issues? Are they within DL or do they offer an
     alternative approach?
  
  **Tags:** deep-learning
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_4 on 2018-03-26. Score: 5 | Accepted answer
  
  To summarize, There are two major issues in applied Deep Learning.
  
  * The first being that computationally , it's exhaustive. Normal CPU's require a lot of time to perform even the basic computation/training with Deep Learning. GPU's are thus recommended however, even they may not be enough in a lot of situations. Typical deep learning models don't support the theoretical time to be in Polynomials. However, if we look at the relatively simpler models in ML for the same tasks, too often we have mathematical guarantees that training time required for such simpler Algorithms is in Polynomials. This, for me, at least is probably the biggest difference.
  
    There're solutions to counter this issue, though. One main approach being is to optimize DL Algorithms to a number of iterations only (instead of looking at the global solutions in practice, just optimize the algorithm to a good local solution, whereas the criterion for "Good" is defined by the user).
  * Another Issue which may be a little bit controversial to young deep learning enthusiasts is that Deep Learning algorithms lack theoretical understanding and reasoning. Deep Neural Networks have been successfully used in a lot of situations including Hand writing recognition, Image processing, Self Driving Cars, Signal Processing, NLP and Biomedical Analysis. In some of these cases, they have even surpassed humans. However, that being said, they're not under any circumstance, theoretically as sound as most of Statistical Methods.
  
    I will not go into detail , rather I leave that up to you. There're pros and cons for every Algorithm/methodology and DL is not an exception. It's very useful as has been proven in a lot of situations and every young Data Scientist must learn at least the basics of DL. However, in the case of relatively simple problems, it's better to use famous Statistical methods as they have a lot of theoretical results/guarantees to support them. Furthermore, from learning point of view, it's always better to start with simple approaches and master them first.
  
  > **Comments:**
  > **username_0**: By 'in polynomials' you mean 'in polynomial time', right? Have you got a reference to support that?  
  > **username_4**: Yes, that's exactly what I mean. Sure, It can be proved in a lot of situations...I will start with the Simplest possible example, Just training a Network with three Nodes, and two layers is NP-Complete problem as shown here.(http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.42.6662&rep=rep1&type=pdf) . Remember that this paper is very old, and now we have more ideas on how to improve in practice, with some heuristics, but still, theoretically, there're no improved results.  
  > **username_4**: Other nice article on the same issue, which also describes some tricks to improve training time in practice. (https://pdfs.semanticscholar.org/9499/140ffba43fd87d84bebc2fc328fa147ac236.pdf)  
  > **username_4**: Let's say, we want to predict the price for something. Simple Linear Regression with least square fit will have a Polynomial time, whereas solving the same issue with Neural Networks (even the simplest of them) will result in NP complete problem. This is a very big difference. Eventually, you have to carefully select an algorithm for a specific task. For example, Least Square fit has specific assumptions, which includes , "The ideal function which the algorithm is learning, can be learned as linear combination of features". If that assumption is not valid,  so are results achieved.  
  > **username_0**: Of course,simply because a problem (in this case, finding optimal weights) is NP-complete does not in itself mean that there aren't efficient practical methods for finding _good_ weights...  
  
  ---
  
  ### Answer by username_1 on 2016-10-13. Score: 5
  
  I have very little experience with ML/DL to call myself either practitioner, but here is my answer on the 1st question:
  
  At its core DL solves well the task of classification. Not every practical problem can be rephrased in terms of classification. Classification domain needs to be known upfront. Although the classification can be applied to any type of data, it's necessary to train the NN with samples of the specific domain where it'll be applied. If the domain is switched at some point, while keeping the same model (NN structure), it'll have to be retrained with new samples. Furthermore, even the best classifiers have "gaps" - [Adversarial Examples](http://www.kdnuggets.com/2015/07/deep-learning-adversarial-examples-misconceptions.html) can be easily constructed from a training sample, such that changes are imperceptible to human, but are misclassified by the trained model.
  
  > **Comments:**
  > **username_0**: 'Classification' can be considered a special case of 'regression', which is probably therefore a better characterization of DL.  
  
  ---
  
  ### Answer by username_2 on 2017-04-04. Score: 3
  
  Question 2.
  I am researching whether Hyper dimensional computing is an alternative to Deep Learning. Hyper-D uses very long bit vectors (10,000 bits) to encode information. The vectors are random and as such they are approximately orthogonal. By grouping and averaging a collection of such vectors a "set" can be formed and later queried to see if an unknown vector belongs to the set. The set can be considered a concept or a generalize image, etc. Training is very fast as is recognition. What needs to be done is simulate the domains in which Deep Learning has been successful and compare Hyper-D with it.
  
  > **Comments:**
  > **username_0**: Interesting. So how does this differ from Kanerva's 'Sparse Distributed Memory'?  
  > **username_2**: Both are developed by Pentti Kanerva.  Look up Hyper dimensional computing to see the difference.  Too long to answer here.  
  
  ---
  
  ### Answer by username_3 on 2017-08-21. Score: 1
  
  From a mathematics point of view one of the major issues in deep networks with several layers are **vanishing** or **unstable gradients**. Each additional hidden layer learns significantly slower, almost nullifying the benefit of the additional layer.
  
  Modern deep learning approaches can improve this behavior, but in simple, old fashioned neural networks this is a well known issue. You can find a well written analysis [here](http://neuralnetworksanddeeplearning.com/chap5.html) for deeper study.
  
  
  ---
  
  