Answers.Body: <p>A little search on Google answers your question.</p>
  <p>XOR input space is not linearly separable. It means that you cannot separate the input points in a 2D space into 1 area and 0 area by simply drawing a line between them. It requires at least 2 lines to separate the XOR input space and consequently 2 output nodes (used as classifiers rather than regression). You can easily find its details in google. Search &quot;XOR problem in Neural Net&quot;.</p>
  <p>You can manually implement the desired Neural Network with two output Nodes acting as classifiers as follows :</p>
  <p><a href="https://i.sstatic.net/KGw32.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KGw32.jpg" alt="Manual Implementation" /></a></p>
  <p>Where A &amp; B are two output Nodes which act as classifiers (by forming AA' and BB' decision lines respectively during training by Backpropagation). The interpretation of the Outputs of the nodes is given in the table where Net column represents the Overall output to be interpreted.</p>
  <p><strong>I showed the above manual implementation just to give you the idea of how classification is done behind the scenes.</strong></p>
  <p>Here is the actual Automatic implementation :</p>
  <p><a href="https://i.sstatic.net/aOumf.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/aOumf.jpg" alt="Practical Implementation" /></a></p>
  <p>Here , all the task is performed by the Neural Net behind the scenes and you get the desired output from the output node in the topmost layer</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-12-31T14:27:01.797Z
Answers.Text: Since you said the 4 required outputs are used as classifiers, why do I need 4 classifiers for a binary output and how to I interpret it.
Answers.UserDisplayName: Miemels
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-12-31T14:42:46.327Z
Answers.Text: Now I'm confused. On the link the network only  has 1 output node. You stated it needs 2. Please explain what I'm getting wrong.
Answers.UserDisplayName: Miemels
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-12-31T14:54:17.787Z
Answers.Text: Ok... Ignore it. I am editing my answer to make you understand.
Answers.UserDisplayName: Programmer
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-12-31T15:20:12.617Z
Answers.Text: Ok thank you very much. I think I understand it now.
Answers.UserDisplayName: Miemels
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-12-31T14:19:13.043Z
Answers.LastActivityDate: 2016-12-31T15:05:49.610Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Programmer
Body: <p>I Build this NN in c++. I reviewed it since 3 days. I checked every line 100 times, but I cant find my error.
  If someone can please help me find the Bugs:
  1. The output is garbage
  2. The weights go from 2e^79 down to -1.8e^80 after approximatly 400 iterations.</p>
  
  <pre><code>mat flip(mat m) {
      mat out(m.n_cols,m.n_rows);
  
      for (int i = 0; i &lt; m.n_rows; ++i)
          for (int j = 0; j &lt; m.n_cols; ++j)
              out(j, i) = m(i, j);
  
      return out;
  }
  
  Layer::Layer(int nodes) :
      rand_engine(time(0))
  {
  
      y = mat (nodes, 1);
      net = mat(nodes, 1);
      e = mat(nodes, 1);
  }
  
  Layer::Layer(int nodes, int next_nodes) :
      Layer(nodes)
  {
  
      this-&gt;next_l = next_l;
  
      auto random = bind(uniform_real_distribution&lt;double&gt;{-1, 1}, rand_engine);
  
      w = mat(next_nodes,nodes);
  
      for (int i = 0; i &lt; w.n_rows; ++i) {
  
          for (int j = 0; j &lt; w.n_cols; ++j) {
              w(i,j) = random();
          }
      }
  }
  
  Layer::Layer(int nodes, Layer* next_l) :
      Layer(nodes,next_l-&gt;y.n_rows)
  {
      this-&gt;next_l = next_l;
  }
  
  void Layer::feed_forward()
  {
      next_l-&gt;net = w*y;
  
  
      for (int i = 0; i &lt; next_l-&gt;y.n_rows;++i) 
          next_l-&gt;y[i] = sig(next_l-&gt;net[i]);
  
  
  }
  
  void Layer::backprop()
  {
  
      for (double d : w)
          cout &lt;&lt; d &lt;&lt; "\t";
  
      e = flip(w)*next_l-&gt;e;
  
  
      for (int i = 0; i &lt; e.n_rows; ++i) {
          e[i] *= net[i] * (1 - net[i]);  
          cout &lt;&lt; e[i] &lt;&lt; '\t';
      }
  
      w += l_rate*(next_l-&gt;e*flip(y));
  
  
  }
  
  void Layer::backprop_last(mat t)
  {
      for (int i = 0; i &lt; e.n_rows; ++i) {
          e[i] = net[i] * (1 - net[i])*(t[i] - y[i]);
          cout &lt;&lt; e[i] &lt;&lt; '\t';
      }
  
  }
  
  void Layer::feed_forward(Layer* next_l)
  {
      this-&gt;next_l = next_l;
  
      feed_forward();
  
  }
  
  double Layer::sig(double x)
  {
      return 1 / (1 + exp(-x));
  }
  
  
  
  
  
  
  Network::Network(vector&lt;int&gt; top):
      top(top)
  {
  
      network = new Layer*[top.size()];
      network[top.size() - 1] = new Layer(top.back());
  
      for (int i = top.size()-2; i &gt; -1; --i)
          network[i] = new Layer(top[i], network[i + 1]);
  
  }
  
  
  Network::~Network()
  {
      delete[] network;
  }
  
  void Network::forward()
  {
      for(int i = 0; i &lt; top.front();++i)
          network[0]-&gt;y[i] = input[i];
  
      for (int i = 0; i &lt; top.size() - 1; ++i)
          network[i]-&gt;feed_forward();
  }
  
  void Network::forward(vector&lt;double&gt; input)
  {
      set_input(input);
      forward();
  }
  
  void Network::backprop()
  {
  
      network[top.size() - 1]-&gt;backprop_last(t_vals);
  
      for (int i = top.size() - 2; i &gt; -1; --i) {
          network[i]-&gt;backprop();
      }
  
  }
  
  void Network::backprop(vector&lt;double&gt; t_vals)
  {
      set_t_vals(t_vals);
      backprop();
  }
  </code></pre>
  
  <p>I know its a bunch of code but im really desprate since I cant find whats wrong. I tested it with a simple XOR.</p>
  
  <p>Edit:
  Heres my Main code:</p>
  
  <pre><code>    #include "Network.h"
  #include &lt;iomanip&gt;
  
  using namespace std;
  
  vector&lt;vector&lt;double&gt;&gt; input = { {0,0},{0,1},{1,1},{1,0} };
  
  vector&lt;vector&lt;double&gt;&gt; true_vals = { {0},{1},{0},{1} };
  
  int main() {
  
      ifstream f("out.txt", fstream::out);
      f.clear();
  
      cout &lt;&lt; fixed;
      cout &lt;&lt; setprecision(5);
  
      Network net({2,5,1});
  
      vector&lt;double&gt; in,t,out; 
  
      auto buf = cout.rdbuf();
  
      for (int i = 0; i &lt; 1000; ++i) {
          cout.rdbuf(f.rdbuf());
  
  
          in = input[i % 4];
  
          net.forward(in);
  
          out = net.get_output();
  
          t = true_vals[i % 4];
  
          net.backprop(t);
          cout &lt;&lt; '\n';
          cout.rdbuf(buf);
          if ((i %101))continue;
  
          cout &lt;&lt; "it: " &lt;&lt; i &lt;&lt; '\n';
  
          cout &lt;&lt; "in:\t";
          for (double d : in)
              cout &lt;&lt; d &lt;&lt; ' ';
  
          cout &lt;&lt; '\n';
  
          cout &lt;&lt; "out:\t";
  
          for (double d : out)
              cout &lt;&lt; d &lt;&lt; ' ';
  
          cout &lt;&lt; '\n';
  
  
          cout &lt;&lt; "true:\t";
  
  
          for (double d : t)
              cout &lt;&lt; d &lt;&lt; ' ';
  
          cout &lt;&lt; '\n';
  
  
          double err = net.get_error();
  
          cout &lt;&lt;"err:\t"&lt;&lt; err &lt;&lt; '\n' &lt;&lt; '\n';
  
      }
  
      cout.rdbuf(NULL);
      f.close();
      return system("pause");
  }
  </code></pre>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-04-17T21:18:46.617Z
Comments.Text: This type of question is more suitable for [Cross Validated](https://stats.stackexchange.com/help/on-topic) or [Data Science](https://datascience.stackexchange.com/help/on-topic).
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-12-31T13:49:58.920Z
Comments.Text: What is this code for? Problem statement?
Comments.UserDisplayName: kiner_shah
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-12-31T13:51:15.597Z
Comments.Text: @kiner_shah that's the code I made. And I want to know if anyone knows why it doesn't work.
Comments.UserDisplayName: Miemels
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-12-31T13:51:45.797Z
Comments.Text: What is that code for? XOR function or AND function or something else? And this is not your entire code! Post your whole code with `main()` function
Comments.UserDisplayName: kiner_shah
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-12-31T13:52:38.640Z
Comments.Text: @kiner_shah it's a simple feed forward and backprop network. Input output and true values are doubles
Comments.UserDisplayName: Miemels
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-12-31T14:06:35.743Z
Comments.Text: @kiner_shah I added the main code
Comments.UserDisplayName: Miemels
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-12-31T14:09:58.573Z
Comments.Text: Ok now I get it, it's an XOR Problem!
Comments.UserDisplayName: kiner_shah
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2016-12-31T18:01:07.760Z
Comments.Text: This question is off-topic here, sorry.
Comments.UserDisplayName: Mithical
Tags: <neural-networks><backpropagation>
Title: Why doesnt my Neural Network work?
UserDisplayName: Miemels
ThreadText: # Why doesnt my Neural Network work?
  
  (Asked by: username_0 on 2016-12-31)
  
  I Build this NN in c++. I reviewed it since 3 days. I checked every line 100 times, but I cant find my error.
  If someone can please help me find the Bugs:
  1. The output is garbage
  2. The weights go from 2e^79 down to -1.8e^80 after approximatly 400 iterations.
  
  ```
  mat flip(mat m) {
      mat out(m.n_cols,m.n_rows);
  
      for (int i = 0; i < m.n_rows; ++i)
          for (int j = 0; j < m.n_cols; ++j)
              out(j, i) = m(i, j);
  
      return out;
  }
  
  Layer::Layer(int nodes) :
      rand_engine(time(0))
  {
  
      y = mat (nodes, 1);
      net = mat(nodes, 1);
      e = mat(nodes, 1);
  }
  
  Layer::Layer(int nodes, int next_nodes) :
      Layer(nodes)
  {
  
      this->next_l = next_l;
  
      auto random = bind(uniform_real_distribution<double>{-1, 1}, rand_engine);
  
      w = mat(next_nodes,nodes);
  
      for (int i = 0; i < w.n_rows; ++i) {
  
          for (int j = 0; j < w.n_cols; ++j) {
              w(i,j) = random();
          }
      }
  }
  
  Layer::Layer(int nodes, Layer* next_l) :
      Layer(nodes,next_l->y.n_rows)
  {
      this->next_l = next_l;
  }
  
  void Layer::feed_forward()
  {
      next_l->net = w*y;
  
      for (int i = 0; i < next_l->y.n_rows;++i) 
          next_l->y[i] = sig(next_l->net[i]);
  
  }
  
  void Layer::backprop()
  {
  
      for (double d : w)
          cout << d << "\t";
  
      e = flip(w)*next_l->e;
  
      for (int i = 0; i < e.n_rows; ++i) {
          e[i] *= net[i] * (1 - net[i]);  
          cout << e[i] << '\t';
      }
  
      w += l_rate*(next_l->e*flip(y));
  
  }
  
  void Layer::backprop_last(mat t)
  {
      for (int i = 0; i < e.n_rows; ++i) {
          e[i] = net[i] * (1 - net[i])*(t[i] - y[i]);
          cout << e[i] << '\t';
      }
  
  }
  
  void Layer::feed_forward(Layer* next_l)
  {
      this->next_l = next_l;
  
      feed_forward();
  
  }
  
  double Layer::sig(double x)
  {
      return 1 / (1 + exp(-x));
  }
  
  Network::Network(vector<int> top):
      top(top)
  {
  
      network = new Layer*[top.size()];
      network[top.size() - 1] = new Layer(top.back());
  
      for (int i = top.size()-2; i > -1; --i)
          network[i] = new Layer(top[i], network[i + 1]);
  
  }
  
  Network::~Network()
  {
      delete[] network;
  }
  
  void Network::forward()
  {
      for(int i = 0; i < top.front();++i)
          network[0]->y[i] = input[i];
  
      for (int i = 0; i < top.size() - 1; ++i)
          network[i]->feed_forward();
  }
  
  void Network::forward(vector<double> input)
  {
      set_input(input);
      forward();
  }
  
  void Network::backprop()
  {
  
      network[top.size() - 1]->backprop_last(t_vals);
  
      for (int i = top.size() - 2; i > -1; --i) {
          network[i]->backprop();
      }
  
  }
  
  void Network::backprop(vector<double> t_vals)
  {
      set_t_vals(t_vals);
      backprop();
  }
  
  ```
  
  I know its a bunch of code but im really desprate since I cant find whats wrong. I tested it with a simple XOR.
  
  Edit:
  Heres my Main code:
  
  ```
      #include "Network.h"
  #include <iomanip>
  
  using namespace std;
  
  vector<vector<double>> input = { {0,0},{0,1},{1,1},{1,0} };
  
  vector<vector<double>> true_vals = { {0},{1},{0},{1} };
  
  int main() {
  
      ifstream f("out.txt", fstream::out);
      f.clear();
  
      cout << fixed;
      cout << setprecision(5);
  
      Network net({2,5,1});
  
      vector<double> in,t,out; 
  
      auto buf = cout.rdbuf();
  
      for (int i = 0; i < 1000; ++i) {
          cout.rdbuf(f.rdbuf());
  
          in = input[i % 4];
  
          net.forward(in);
  
          out = net.get_output();
  
          t = true_vals[i % 4];
  
          net.backprop(t);
          cout << '\n';
          cout.rdbuf(buf);
          if ((i %101))continue;
  
          cout << "it: " << i << '\n';
  
          cout << "in:\t";
          for (double d : in)
              cout << d << ' ';
  
          cout << '\n';
  
          cout << "out:\t";
  
          for (double d : out)
              cout << d << ' ';
  
          cout << '\n';
  
          cout << "true:\t";
  
          for (double d : t)
              cout << d << ' ';
  
          cout << '\n';
  
          double err = net.get_error();
  
          cout <<"err:\t"<< err << '\n' << '\n';
  
      }
  
      cout.rdbuf(NULL);
      f.close();
      return system("pause");
  }
  
  ```
  
  **Tags:** neural-networks, backpropagation
  
  > **Comments:**
  > **username_2**: What is this code for? Problem statement?  
  > **username_0**: @username_2 that's the code I made. And I want to know if anyone knows why it doesn't work.  
  > **username_2**: What is that code for? XOR function or AND function or something else? And this is not your entire code! Post your whole code with `main()` function  
  > **username_0**: @username_2 it's a simple feed forward and backprop network. Input output and true values are doubles  
  > **username_0**: @username_2 I added the main code  
  > **username_2**: Ok now I get it, it's an XOR Problem!  
  > **username_3**: This question is off-topic here, sorry.  
  > **username_1**: This type of question is more suitable for [Cross Validated](https://stats.stackexchange.com/help/on-topic) or [Data Science](https://datascience.stackexchange.com/help/on-topic).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2016-12-31. Score: 2
  
  A little search on Google answers your question.
  
  XOR input space is not linearly separable. It means that you cannot separate the input points in a 2D space into 1 area and 0 area by simply drawing a line between them. It requires at least 2 lines to separate the XOR input space and consequently 2 output nodes (used as classifiers rather than regression). You can easily find its details in google. Search "XOR problem in Neural Net".
  
  You can manually implement the desired Neural Network with two output Nodes acting as classifiers as follows :
  
  [![Manual Implementation](https://i.sstatic.net/KGw32.jpg)](https://i.sstatic.net/KGw32.jpg)
  
  Where A & B are two output Nodes which act as classifiers (by forming AA' and BB' decision lines respectively during training by Backpropagation). The interpretation of the Outputs of the nodes is given in the table where Net column represents the Overall output to be interpreted.
  
  **I showed the above manual implementation just to give you the idea of how classification is done behind the scenes.**
  
  Here is the actual Automatic implementation :
  
  [![Practical Implementation](https://i.sstatic.net/aOumf.jpg)](https://i.sstatic.net/aOumf.jpg)
  
  Here , all the task is performed by the Neural Net behind the scenes and you get the desired output from the output node in the topmost layer
  
  > **Comments:**
  > **username_0**: Since you said the 4 required outputs are used as classifiers, why do I need 4 classifiers for a binary output and how to I interpret it.  
  > **username_0**: Now I'm confused. On the link the network only  has 1 output node. You stated it needs 2. Please explain what I'm getting wrong.  
  > **username_4**: Ok... Ignore it. I am editing my answer to make you understand.  
  > **username_0**: Ok thank you very much. I think I understand it now.  
  
  ---
  
  