Answers.Body: <p>Ideally neural networks should be able to find out the function out on it's own without us providing the spherical features. After some experimentation I was able to reach a configuration where we do not need anything except <span class="math-container">$X_1$</span> and <span class="math-container">$X_2$</span>. This net converged after about 1500 epochs which is quite long. So the best way might still be to add additional features but I am just trying to say that it is still possible to converge without them.</p>
  
  <p><img src="https://i.sstatic.net/saZat.jpg" alt="Solution without additional features"></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-04-27T12:51:09.063Z
Answers.LastActivityDate: 2018-12-15T20:34:09.603Z
Answers.LastEditDate: 2018-12-15T20:34:09.603Z
Answers.UserDisplayName: Dheeraj Pb
Answers.Body: <p>By cheating... <code>theta</code> is <span class="math-container">$\arctan(y,x)$</span>, <span class="math-container">$r$</span> is <span class="math-container">$\sqrt{(x^2 + y^2)}$</span>.</p>
  
  <p>In theory, <span class="math-container">$x^2$</span> and <span class="math-container">$y^2$</span> should work, but, in practice, they somehow failed, even though, occasionally, it works.</p>
  
  <p><a href="https://i.sstatic.net/mT4Nc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/mT4Nc.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-26T23:41:45.643Z
Answers.Text: Can you elaborate on how you "cheated"? How did you add these features? Did you download the playground from GitHub and modify it? Or is there a more direct way to do this?
Answers.UserDisplayName: Jim
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-10T16:39:37.593Z
Answers.Text: I was wondering how to apply this hack too. Tried to do it inside browser but had no luck with that. Then I decided to fork the source code and added the new input features. You can see it [here](https://github.com/cankut/playground/commit/b982c86e0d89f42b68fcda8be70cdc78df56583f)
Answers.UserDisplayName: Cankut
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-15T07:59:04.600Z
Answers.LastActivityDate: 2019-02-26T17:24:24.193Z
Answers.LastEditDate: 2019-02-26T17:24:24.193Z
Answers.UserDisplayName: anonisnotanon
Answers.Body: <p>You can increase no of hidden layers. Following is an example (But not very efficient)</p>
  <p><a href="https://i.sstatic.net/W3TPj.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/W3TPj.jpg" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-20T03:31:27.763Z
Answers.LastActivityDate: 2020-07-20T03:31:27.763Z
Answers.UserDisplayName: Kartheek Kumar
Answers.Body: <p>May be you need reset all settings, and select x squared and y squared, only 1 hidden layer with 5 neurons.</p>
  <p><a href="https://i.sstatic.net/UeIKW.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/UeIKW.png" alt="With 5 neurons, x squared and y squared on" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-05T07:18:07.313Z
Answers.LastActivityDate: 2021-06-05T07:18:07.313Z
Answers.UserDisplayName: Jorgenio Selenio
Answers.Body: <p>May be you need reset all settings, only 1 hidden layer with 8 neurons, and 90% Dataset</p>
  <p><a href="https://i.sstatic.net/fzOoAZN6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fzOoAZN6.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-08T11:50:12.203Z
Answers.LastActivityDate: 2024-11-08T11:50:12.203Z
Answers.UserDisplayName: Muhammad Numan
Answers.Body: <p><a href="https://i.sstatic.net/0FQY9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/0FQY9.png" alt="vanilla tensorflow playground spiral solution"></a></p>
  
  <p>This is an example of vanilla Tensorflow playground with no added features and no modifications.
  The run for Spiral was between 187 to ~300 Epoch, depending. 
  I used Lasso Regularization L1 so I could eliminate coefficients. 
  I decreased the batch size by 1 to keep the output from over fitting. 
  In my second example I added some noise to the data set then upped the L1 to compensate. </p>
  
  <p><a href="https://i.sstatic.net/OXPkd.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/OXPkd.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-28T12:21:43.583Z
Answers.LastActivityDate: 2019-06-28T12:21:43.583Z
Answers.UserDisplayName: Jamin Quimby
Answers.Body: <p><a href="https://i.sstatic.net/rh1ch.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rh1ch.jpg" alt="Spiral Dataset Classification with Test Loss=0.000 using Playground Tensor flow" /></a></p>
  <p>This is the architecture proposed and tested on the playground tensor flow for the Spiral Dataset. Two Hidden Layers with 8 neurons each is proposed with Tanh activation function.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-25T12:43:50.003Z
Answers.LastActivityDate: 2021-05-25T12:46:19.310Z
Answers.LastEditDate: 2021-05-25T12:46:19.310Z
Answers.UserDisplayName: Muhammad Hamza Zafar
Answers.Body: <p><a href="https://i.sstatic.net/gw3P3P1I.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gw3P3P1I.png" alt="enter image description here" /></a>I’ve experimented with multilayer perceptrons (MLPs) to understand their efficacy in handling complex, non-linear patterns like spirals. Here are some insights and configurations that I’ve found beneficial:</p>
  <pre><code>1.  Layer Configuration: Using multiple hidden layers helps in capturing the intricacies of spiral boundaries. Typically, starting with two layers and adjusting based on validation accuracy is a good approach.
  2.  Neuron Distribution: Increasing the number of neurons in the first hidden layer and gradually reducing it in subsequent layers can effectively capture the finer details and variations in the data.
  3.  Activation Functions: Non-linear activation functions like ReLU have shown promising results in enhancing model capabilities to learn complex patterns over traditional sigmoid functions, due to their ability to mitigate the vanishing gradient problem.
  4.  Training Techniques: Employ techniques like batch normalization and dropout to improve generalization and prevent overfitting, which is crucial given the complex nature of spiral data.
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-17T11:48:14.623Z
Answers.LastActivityDate: 2024-09-17T11:51:13.633Z
Answers.LastEditDate: 2024-09-17T11:51:13.633Z
Answers.UserDisplayName: Novalis133
Answers.Body: <p>The <a href="https://playground.tensorflow.org/#activation=tanh&amp;batchSize=5&amp;dataset=spiral&amp;regDataset=reg-plane&amp;learningRate=0.03&amp;regularizationRate=0&amp;noise=0&amp;networkShape=5,4&amp;seed=0.84823&amp;showTestData=false&amp;discretize=true&amp;percTrainData=50&amp;x=true&amp;y=true&amp;xTimesY=false&amp;xSquared=false&amp;ySquared=false&amp;cosX=false&amp;sinX=true&amp;cosY=false&amp;sinY=true&amp;collectStats=false&amp;problem=classification&amp;initZero=false&amp;hideText=false&amp;regularization_hide=false&amp;numHiddenLayers_hide=false" rel="nofollow noreferrer">solution</a> I reached after an hour of trial usually <strong>converges in just 100 epochs</strong>. </p>
  
  <p>Yeah, I know it does not have the smoothest decision boundary out there, but it converges pretty fast.</p>
  
  <p><a href="https://i.sstatic.net/BZLo6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/BZLo6.png" alt="![0.6% test error"></a></p>
  
  <p>I learned a few things from this spiral experiment:-</p>
  
  <ul>
  <li><strong>The output layer should be greater than or equal to the input layer</strong>. At least that's what I noticed in the case of this spiral problem.</li>
  <li><strong>Keep the initial learning rate high</strong>, like 0.1 in this case, then as you approach a low test error like 3-5% or less, decrease the learning rate by a notch(0.03) or two. This helps in converging faster and avoids jumping around the global minima.</li>
  <li>You can see the effects of keeping the learning rate high by checking the error graph at the top right.</li>
  <li>For smaller batch sizes like 1, 0.1 is too high a learning rate as the model fails to converge as it jumps around the global minima.</li>
  <li>So, if you would like to keep a high learning rate(0.1), keep the batch size high(10) as well. This usually gives a slow yet smoother convergence.</li>
  </ul>
  
  <p>Coincidentally the solution I came up with is very similar to the one provided by <a href="https://ai.stackexchange.com/users/2492/salvador-dali">Salvador Dali</a>.</p>
  
  <p>Kindly add a comment, if you find any more intuitions or reasonings.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-05T15:56:34.683Z
Answers.LastActivityDate: 2020-03-05T18:12:23.343Z
Answers.LastEditDate: 2020-03-05T18:12:23.343Z
Answers.UserDisplayName: dracarys3
Answers.Body: <p>There are many approaches to this kind of problem. The most obvious one is to <strong>create new features</strong>. The best features I can come up with is to transform the coordinates to <a href="https://en.wikipedia.org/wiki/Spherical_coordinate_system" rel="noreferrer">spherical coordinates</a>. </p>
  
  <p>I have not found a way to do it in playground, so I just created a few features that should help with this (sin features). After <a href="http://playground.tensorflow.org/#activation=tanh&amp;batchSize=10&amp;dataset=spiral&amp;regDataset=reg-plane&amp;learningRate=0.03&amp;regularizationRate=0&amp;noise=0&amp;networkShape=4,2&amp;seed=0.73263&amp;showTestData=false&amp;discretize=false&amp;percTrainData=50&amp;x=true&amp;y=true&amp;xTimesY=false&amp;xSquared=false&amp;ySquared=false&amp;cosX=false&amp;sinX=true&amp;cosY=false&amp;sinY=true&amp;collectStats=false&amp;problem=classification&amp;initZero=false&amp;hideText=false" rel="noreferrer">500 iterations</a> it will saturate and will fluctuate at 0.1 score. This suggest that no further improvement will be done and most probably I should make the hidden layer wider or add another layer.</p>
  
  <p>Not a surprise that after adding <a href="http://playground.tensorflow.org/#activation=tanh&amp;batchSize=10&amp;dataset=spiral&amp;regDataset=reg-plane&amp;learningRate=0.03&amp;regularizationRate=0&amp;noise=0&amp;networkShape=5,2&amp;seed=0.73263&amp;showTestData=false&amp;discretize=false&amp;percTrainData=50&amp;x=true&amp;y=true&amp;xTimesY=false&amp;xSquared=false&amp;ySquared=false&amp;cosX=false&amp;sinX=true&amp;cosY=false&amp;sinY=true&amp;collectStats=false&amp;problem=classification&amp;initZero=false&amp;hideText=false" rel="noreferrer">just one neuron to the hidden layer</a> you easily get 0.013 after 300 iterations. Similar thing happens by adding a new layer (0.017, but after significantly longer 500 iterations. Also no surprise as it is harder to propagate the errors). Most probably you can play with a learning rate or do an adaptive learning to make it faster, but this is not the point here.</p>
  
  <p><a href="https://i.sstatic.net/tck2s.png" rel="noreferrer"><img src="https://i.sstatic.net/tck2s.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-09-18T23:27:57.760Z
Answers.LastActivityDate: 2016-09-18T23:27:57.760Z
Answers.UserDisplayName: Salvador Dali
Body: <p>I have been messing around in <a href="http://playground.tensorflow.org/#activation=tanh&amp;batchSize=10&amp;dataset=spiral&amp;regDataset=reg-plane&amp;learningRate=0.03&amp;regularizationRate=0&amp;noise=0&amp;networkShape=4,2&amp;seed=0.73263&amp;showTestData=false&amp;discretize=false&amp;percTrainData=50&amp;x=true&amp;y=true&amp;xTimesY=false&amp;xSquared=false&amp;ySquared=false&amp;cosX=false&amp;sinX=false&amp;cosY=false&amp;sinY=false&amp;collectStats=false&amp;problem=classification&amp;initZero=false&amp;hideText=false" rel="noreferrer">tensorflow playground</a>. One of the input data sets is a spiral. No matter what input parameters I choose, no matter how wide and deep the neural network I make, I cannot fit the spiral. How do data scientists fit data of this shape?</p>
  
Tags: <neural-networks><machine-learning><tensorflow><regression>
Title: How to classify data which is spiral in shape?
UserDisplayName: Souradeep Nanda
ThreadText: # How to classify data which is spiral in shape?
  
  (Asked by: username_0 on 2016-09-18)
  
  I have been messing around in [tensorflow playground](http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.73263&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false). One of the input data sets is a spiral. No matter what input parameters I choose, no matter how wide and deep the neural network I make, I cannot fit the spiral. How do data scientists fit data of this shape?
  
  **Tags:** neural-networks, machine-learning, tensorflow, regression
  
  
  ---
  
  ## 10 answers
  
  ### Answer by username_12 on 2016-09-18. Score: 16 | Accepted answer
  
  There are many approaches to this kind of problem. The most obvious one is to **create new features**. The best features I can come up with is to transform the coordinates to [spherical coordinates](https://en.wikipedia.org/wiki/Spherical_coordinate_system).
  
  I have not found a way to do it in playground, so I just created a few features that should help with this (sin features). After [500 iterations](http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,2&seed=0.73263&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification&initZero=false&hideText=false) it will saturate and will fluctuate at 0.1 score. This suggest that no further improvement will be done and most probably I should make the hidden layer wider or add another layer.
  
  Not a surprise that after adding [just one neuron to the hidden layer](http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=5,2&seed=0.73263&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification&initZero=false&hideText=false) you easily get 0.013 after 300 iterations. Similar thing happens by adding a new layer (0.017, but after significantly longer 500 iterations. Also no surprise as it is harder to propagate the errors). Most probably you can play with a learning rate or do an adaptive learning to make it faster, but this is not the point here.
  
  [![enter image description here](https://i.sstatic.net/tck2s.png)](https://i.sstatic.net/tck2s.png)
  
  
  ---
  
  ### Answer by username_1 on 2018-04-27. Score: 9
  
  Ideally neural networks should be able to find out the function out on it's own without us providing the spherical features. After some experimentation I was able to reach a configuration where we do not need anything except $X\_1$ and $X\_2$. This net converged after about 1500 epochs which is quite long. So the best way might still be to add additional features but I am just trying to say that it is still possible to converge without them.
  
  ![Solution without additional features](https://i.sstatic.net/saZat.jpg)
  
  
  ---
  
  ### Answer by username_2 on 2019-01-15. Score: 5
  
  By cheating... `theta` is $\arctan(y,x)$, $r$ is $\sqrt{(x^2 + y^2)}$.
  
  In theory, $x^2$ and $y^2$ should work, but, in practice, they somehow failed, even though, occasionally, it works.
  
  [![enter image description here](https://i.sstatic.net/mT4Nc.png)](https://i.sstatic.net/mT4Nc.png)
  
  > **Comments:**
  > **username_3**: Can you elaborate on how you "cheated"? How did you add these features? Did you download the playground from GitHub and modify it? Or is there a more direct way to do this?  
  > **username_4**: I was wondering how to apply this hack too. Tried to do it inside browser but had no luck with that. Then I decided to fork the source code and added the new input features. You can see it [here](https://github.com/username_4/playground/commit/b982c86e0d89f42b68fcda8be70cdc78df56583f)  
  
  ---
  
  ### Answer by username_5 on 2020-07-20. Score: 1
  
  You can increase no of hidden layers. Following is an example (But not very efficient)
  
  [![enter image description here](https://i.sstatic.net/W3TPj.jpg)](https://i.sstatic.net/W3TPj.jpg)
  
  
  ---
  
  ### Answer by username_6 on 2021-06-05. Score: 0
  
  May be you need reset all settings, and select x squared and y squared, only 1 hidden layer with 5 neurons.
  
  [![With 5 neurons, x squared and y squared on](https://i.sstatic.net/UeIKW.png)](https://i.sstatic.net/UeIKW.png)
  
  
  ---
  
  ### Answer by username_7 on 2024-11-08. Score: 0
  
  May be you need reset all settings, only 1 hidden layer with 8 neurons, and 90% Dataset
  
  [![enter image description here](https://i.sstatic.net/fzOoAZN6.png)](https://i.sstatic.net/fzOoAZN6.png)
  
  
  ---
  
  ### Answer by username_8 on 2019-06-28. Score: 0
  
  [![vanilla tensorflow playground spiral solution](https://i.sstatic.net/0FQY9.png)](https://i.sstatic.net/0FQY9.png)
  
  This is an example of vanilla Tensorflow playground with no added features and no modifications.
  The run for Spiral was between 187 to ~300 Epoch, depending.
  I used Lasso Regularization L1 so I could eliminate coefficients.
  I decreased the batch size by 1 to keep the output from over fitting.
  In my second example I added some noise to the data set then upped the L1 to compensate.
  
  [![enter image description here](https://i.sstatic.net/OXPkd.png)](https://i.sstatic.net/OXPkd.png)
  
  
  ---
  
  ### Answer by username_9 on 2021-05-25. Score: 0
  
  [![Spiral Dataset Classification with Test Loss=0.000 using Playground Tensor flow](https://i.sstatic.net/rh1ch.jpg)](https://i.sstatic.net/rh1ch.jpg)
  
  This is the architecture proposed and tested on the playground tensor flow for the Spiral Dataset. Two Hidden Layers with 8 neurons each is proposed with Tanh activation function.
  
  
  ---
  
  ### Answer by username_10 on 2024-09-17. Score: 0
  
  [![enter image description here](https://i.sstatic.net/gw3P3P1I.png)](https://i.sstatic.net/gw3P3P1I.png)I’ve experimented with multilayer perceptrons (MLPs) to understand their efficacy in handling complex, non-linear patterns like spirals. Here are some insights and configurations that I’ve found beneficial:
  
  ```
  1.  Layer Configuration: Using multiple hidden layers helps in capturing the intricacies of spiral boundaries. Typically, starting with two layers and adjusting based on validation accuracy is a good approach.
  2.  Neuron Distribution: Increasing the number of neurons in the first hidden layer and gradually reducing it in subsequent layers can effectively capture the finer details and variations in the data.
  3.  Activation Functions: Non-linear activation functions like ReLU have shown promising results in enhancing model capabilities to learn complex patterns over traditional sigmoid functions, due to their ability to mitigate the vanishing gradient problem.
  4.  Training Techniques: Employ techniques like batch normalization and dropout to improve generalization and prevent overfitting, which is crucial given the complex nature of spiral data.
  
  ```
  
  
  ---
  
  ### Answer by username_11 on 2020-03-05. Score: 0
  
  The [solution](https://playground.tensorflow.org/#activation=tanh&batchSize=5&dataset=spiral&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=5,4&seed=0.84823&showTestData=false&discretize=true&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification&initZero=false&hideText=false&regularization_hide=false&numHiddenLayers_hide=false) I reached after an hour of trial usually **converges in just 100 epochs**.
  
  Yeah, I know it does not have the smoothest decision boundary out there, but it converges pretty fast.
  
  [![![0.6% test error](https://i.sstatic.net/BZLo6.png)](https://i.sstatic.net/BZLo6.png)
  
  I learned a few things from this spiral experiment:-
  
  * **The output layer should be greater than or equal to the input layer**. At least that's what I noticed in the case of this spiral problem.
  * **Keep the initial learning rate high**, like 0.1 in this case, then as you approach a low test error like 3-5% or less, decrease the learning rate by a notch(0.03) or two. This helps in converging faster and avoids jumping around the global minima.
  * You can see the effects of keeping the learning rate high by checking the error graph at the top right.
  * For smaller batch sizes like 1, 0.1 is too high a learning rate as the model fails to converge as it jumps around the global minima.
  * So, if you would like to keep a high learning rate(0.1), keep the batch size high(10) as well. This usually gives a slow yet smoother convergence.
  
  Coincidentally the solution I came up with is very similar to the one provided by [username_12](https://ai.stackexchange.com/users/2492/username_12-username_12).
  
  Kindly add a comment, if you find any more intuitions or reasonings.
  
  
  ---
  
  