Answers.Body: <p>There are two factors that will change the ability of a deep neural network to fit a given dataset: either you need more data, or a deeper and wider network. Since the pattern is only 2-d, it can likely be approximated by some sort of simple periodic function. A DNN can approximate periodic functions pretty well, so the issue is probably that you don't have enough data. </p>
  
  <p>If you have an apriori belief that the pattern is well approximated by K-nearest neighbors, then you could do the following:</p>
  
  <ol>
  <li>Fit a K-NN model to the data.</li>
  <li>Generate <span class="math-container">$N$</span> new points uniformly at random from the input space.</li>
  <li>Label the <span class="math-container">$N$</span> new points using the K-NN model.</li>
  <li>Fit your DNN to the original dataset, plus the <span class="math-container">$N$</span> new points.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-28T00:57:45.797Z
Answers.LastActivityDate: 2019-09-28T00:57:45.797Z
Answers.UserDisplayName: John Doucette
Body: <p>The following X-shape alternated pattern can be separated quite well and super fast by K-nearest Neighbour algorithm (go to <a href="https://ml-playground.com" rel="nofollow noreferrer">https://ml-playground.com</a> to test it):</p>
  <p><a href="https://i.sstatic.net/vsi8v.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vsi8v.png" alt="image of x-shaped data" /></a></p>
  <p>However, DNN seems to face great struggles to separate that X-shape alternated data. <strong>Is it possible to do K-nearest before DNN</strong>, ie. set the DNN weights somehow to simulate the result of K-nearest before doing DNN training?</p>
  <p>Another place to test the X-shape alternated data: <a href="https://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html" rel="nofollow noreferrer">https://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-26T05:56:08.763Z
Comments.Text: If data already labeled clustering wouldn't add  anything new. The only known way to set weights for DNN is training form random initialization. The convergence of training is based on randomness of weights (it's used in all existing proofs). It could be taht in future there will be non-random way to initialize weights, but those method necessarily will be very complex and putting additional constraint on them probably would be extremely hard. Anyway there is no such non-random methods for now.
Comments.UserDisplayName: mirror2image
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-26T06:04:57.897Z
Comments.Text: yeah, but if we can just initialise weights in some greedy manner
Comments.UserDisplayName: Dan D.
Tags: <neural-networks><deep-learning><learning-algorithms><greedy-ai><symbolic-ai>
Title: Is it possible to do K-nearest-neighbours before training DNN
UserDisplayName: Dan D.
ThreadText: # Is it possible to do K-nearest-neighbours before training DNN
  
  (Asked by: username_0 on 2019-09-26)
  
  The following X-shape alternated pattern can be separated quite well and super fast by K-nearest Neighbour algorithm (go to <https://ml-playground.com> to test it):
  
  [![image of x-shaped data](https://i.sstatic.net/vsi8v.png)](https://i.sstatic.net/vsi8v.png)
  
  However, DNN seems to face great struggles to separate that X-shape alternated data. **Is it possible to do K-nearest before DNN**, ie. set the DNN weights somehow to simulate the result of K-nearest before doing DNN training?
  
  Another place to test the X-shape alternated data: <https://cs.stanford.edu/people/karpathy/convnetjs/demo/classify2d.html>
  
  **Tags:** neural-networks, deep-learning, learning-algorithms, greedy-ai, symbolic-ai
  > **Comments:**
  > **username_1**: If data already labeled clustering wouldn't add  anything new. The only known way to set weights for DNN is training form random initialization. The convergence of training is based on randomness of weights (it's used in all existing proofs). It could be taht in future there will be non-random way to initialize weights, but those method necessarily will be very complex and putting additional constraint on them probably would be extremely hard. Anyway there is no such non-random methods for now.  
  > **username_0**: yeah, but if we can just initialise weights in some greedy manner  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-09-28. Score: 2 | Accepted answer
  
  There are two factors that will change the ability of a deep neural network to fit a given dataset: either you need more data, or a deeper and wider network. Since the pattern is only 2-d, it can likely be approximated by some sort of simple periodic function. A DNN can approximate periodic functions pretty well, so the issue is probably that you don't have enough data.
  
  If you have an apriori belief that the pattern is well approximated by K-nearest neighbors, then you could do the following:
  
  1. Fit a K-NN model to the data.
  2. Generate $N$ new points uniformly at random from the input space.
  3. Label the $N$ new points using the K-NN model.
  4. Fit your DNN to the original dataset, plus the $N$ new points.
  
  
  ---
  
  