Answers.Body: <p>In neural networks, the family of functions and the shapes that they can make for decision surfaces is determined by the activation function you use (in your case, <code>tanh</code> or hyperbolic tangent). </p>
  
  <p>Assuming at least one hidden layer, then the <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">universal approximation theorem</a> applies. How closely you can approximate any given function is limited by the number of neurons, and not strongly by the choice of activation function. However, the choice of activation function is still relevant to how good an approximation is. When you get to low numbers of neurons in a hidden layer, then the approximations are more strongly tied to the nature of the activation function.</p>
  
  <p>With one neuron in a hidden layer, you can only approximate some affine transformation of the activation function. Other low numbers, such as 2, 3 etc, will still show strong tendencies for certain families of shapes. This is very similar conceptually to using limited number of frequencies in a Fourier transform - if you limit yourself to only <span class="math-container">$a_1 \text{sin}(x) + a_2 \text{sin}(2x)$</span> to approximate a function, then you will definitely notice the sinusoidal building blocks in any output.</p>
  
  <p>I suspect that if you changed the activation function in the first hidden layer to <span class="math-container">$f(x) = x^2$</span> then you could get a good result with two neurons per layer. If you then took that network and tried to train it on a simple linear split, it would fail, always producing some curved closed surface that covered the training examples as best that it could - kind of the opposite problem as you are seeing with your circular pattern fitted to NN with tanh activations throughout.</p>
  
  <p>One interesting thing about using <span class="math-container">$f(x) = x^2$</span> is that this is a deliberate choice (given knowledge about how you constructed the example) to map your input space to a new space where examples can be linearly separated. In fact this appears to be what layers in multi-layer NNs learn - each layer incrementally and progressively maps its input space to a new space where examples can be better separated in a linear fashion.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-22T08:55:57.437Z
Answers.Text: You sufficiently covered my question, but I will wait a little bit until I mark this answer as the correct one. I will try to use x^2
Answers.UserDisplayName: entropyfeverone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-22T08:57:29.850Z
Answers.Text: @entropyfeverone: Thank you. In general, $x^2$ makes a very bad activation function, it is only useful here because you already know the correct distribution can be modelled using it. The more usual thing to do in practice is to have sufficiently large layers.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-22T09:50:34.433Z
Answers.Text: It worked :) Thank you. You can check the result in my question.
Answers.UserDisplayName: entropyfeverone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-22T08:38:08.150Z
Answers.LastActivityDate: 2020-03-22T08:47:07.437Z
Answers.LastEditDate: 2020-03-22T08:47:07.437Z
Answers.UserDisplayName: Neil Slater
Body: <p><strong>[TL;DR]</strong>  </p>
  
  <p>I generated two classes Red and Blue on a 2D space. Red are points on Unit Circle and Blue are points on a Circle Ring with radius limits (3,4). I tried to train a Multi Layer Perceptron with different number of hidden layers, <strong>BUT</strong> all the hidden layers had 2 neurons. The MLP never reached 100% accuracy. I tried to visualize how the MLP would classify the points of the 2D space with Black and White. This is the final image I get:</p>
  
  <p><a href="https://i.sstatic.net/69rSY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/69rSY.png" alt="enter image description here"></a></p>
  
  <p>At first, I was expecting that the MLP could classify 2 classes on a 2D space with 2 Neurons at each hidden layer, and I was expecting to see a white circle encapsulating the red points and the rest be a black space. Is there a (mathematical) reason, why the MLP fails to create a close shape, rather it seems to go from infinity to infinity on a 2d space ?? (Notice: If I use 3 neurons at each hidden layer, the MLP succeeds quite fast). </p>
  
  <p><strong>[Notebook Style]</strong>  </p>
  
  <p>I generated two classes Red and Blue on a 2D space.<br>
  Red are points on Unit Circle </p>
  
  <pre><code>size_ = 200
  classA_r = np.random.uniform(low = 0, high = 1, size = size_)
  classA_theta = np.random.uniform(low = 0, high = 2*np.pi, size = size_)
  classA_x = classA_r * np.cos(classA_theta)
  classA_y = classA_r * np.sin(classA_theta)
  
  </code></pre>
  
  <p>and Blue are points on a Circle Ring with radius limits (3,4).  </p>
  
  <pre><code>classB_r = np.random.uniform(low = 2, high = 3, size = size_)
  classB_theta = np.random.uniform(low = 0, high = 2*np.pi, size = size_)
  classB_x = classB_r * np.cos(classB_theta)
  classB_y = classB_r * np.sin(classB_theta)
  </code></pre>
  
  <p>I tried to train a Multi Layer Perceptron with different number of hidden layers, <strong>BUT</strong> all the hidden layers had 2 neurons. </p>
  
  <pre><code>hidden_layers = 15
  inputs = Input(shape=(2,))
  dnn = inputs
  for l_no in range(hidden_layers):
      dnn = Dense(2, activation='tanh', name = "layer_{}".format(l_no))(dnn)
  outputs = Dense(2, activation='softmax', name = "layer_out")(dnn)
  
  model = Model(inputs=inputs, outputs=outputs)
  
  model.compile(optimizer='adam', loss='categorical_crossentropy', metrics='accuracy'])
  </code></pre>
  
  <p>The MLP never reached 100% accuracy. I tried to visualize how the MLP would classify the points of the 2D space with Black and White.</p>
  
  <pre><code>limit = 4
  step = 0.2
  grid = []
  x = -limit
  while x &lt;= limit:
      y = -limit
      while y &lt;= limit:
          grid.append([x, y])
          y += step
      x += step
  grid = np.array(grid)
  </code></pre>
  
  <pre><code>prediction = model.predict(grid)
  </code></pre>
  
  <p>This is the final image I get:</p>
  
  <pre><code>xs = []
  ys = []
  cs = []
  for point in grid:
      xs.append(point[0])
      ys.append(point[1])
  for pred in prediction:
      cs.append(pred[0])
  
  plt.scatter(xs, ys, c = cs, s=70, cmap = 'gray')
  plt.scatter(classA_x, classA_y, c = 'r', s= 50)
  plt.scatter(classB_x, classB_y, c = 'b', s= 50)
  plt.show()
  </code></pre>
  
  <p><a href="https://i.sstatic.net/69rSY.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/69rSY.png" alt="enter image description here"></a></p>
  
  <p>At first, I was expecting that the MLP could classify 2 classes on a 2D space with 2 Neurons at each hidden layer, and I was expecting to see a white circle encapsulating the red points and the rest be a black space. Is there a (mathematical) reason, why the MLP fails to create a close shape, rather it seems to go from infinity to infinity on a 2d space ?? (Notice: If I use 3 neurons at each hidden layer, the MLP succeeds quite fast).</p>
  
  <p>What I mean by a closed shape, take a look at the second image which was generated by using 3 neurons at each layer:</p>
  
  <pre><code>for l_no in range(hidden_layers):
      dnn = Dense(3, activation='tanh', name = "layer_{}".format(l_no))(dnn)
  </code></pre>
  
  <p><a href="https://i.sstatic.net/SunX6.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/SunX6.png" alt="enter image description here"></a> </p>
  
  <p><strong><em>[According to Marked Answer]</em></strong>  </p>
  
  <pre><code>from keras import backend as K
  def x_squared(x):
      x = K.abs(x) * K.abs(x)
      return x
  hidden_layers = 3
  inputs = Input(shape=(2,))
  dnn = inputs
  for l_no in range(hidden_layers):
      dnn = Dense(2, activation=x_squared, name = "layer_{}".format(l_no))(dnn)
  outputs = Dense(2, activation='softsign', name = "layer_out")(dnn)
  model.compile(optimizer='adam',
                loss='mean_squared_error',
                metrics=['accuracy'])
  </code></pre>
  
  <p>I get:</p>
  
  <p><a href="https://i.sstatic.net/9bKSC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9bKSC.png" alt="enter image description here"></a></p>
  
Tags: <machine-learning><keras><multilayer-perceptrons><linear-algebra>
Title: Why MLP cannot approximate a closed shape function?
UserDisplayName: entropyfeverone
ThreadText: # Why MLP cannot approximate a closed shape function?
  
  (Asked by: username_0 on 2020-03-22)
  
  [TL;DR] 
  I generated two classes Red and Blue on a 2D space. Red are points on Unit Circle and Blue are points on a Circle Ring with radius limits (3,4). I tried to train a Multi Layer Perceptron with different number of hidden layers, BUT all the hidden layers had 2 neurons. The MLP never reached 100% accuracy. I tried to visualize how the MLP would classify the points of the 2D space with Black and White. This is the final image I get:
  
  At first, I was expecting that the MLP could classify 2 classes on a 2D space with 2 Neurons at each hidden layer, and I was expecting to see a white circle encapsulating the red points and the rest be a black space. Is there a (mathematical) reason, why the MLP fails to create a close shape, rather it seems to go from infinity to infinity on a 2d space ?? (Notice: If I use 3 neurons at each hidden layer, the MLP succeeds quite fast). 
  [Notebook Style] 
  I generated two classes Red and Blue on a 2D space.
  Red are points on Unit Circle 
  size_ = 200
  classA_r = np.random.uniform(low = 0, high = 1, size = size_)
  classA_theta = np.random.uniform(low = 0, high = 2*np.pi, size = size_)
  classA_x = classA_r * np.cos(classA_theta)
  classA_y = classA_r * np.sin(classA_theta)
  
  
  and Blue are points on a Circle Ring with radius limits (3,4).  
  classB_r = np.random.uniform(low = 2, high = 3, size = size_)
  classB_theta = np.random.uniform(low = 0, high = 2*np.pi, size = size_)
  classB_x = classB_r * np.cos(classB_theta)
  classB_y = classB_r * np.sin(classB_theta)
  
  I tried to train a Multi Layer Perceptron with different number of hidden layers, BUT all the hidden layers had 2 neurons. 
  hidden_layers = 15
  inputs = Input(shape=(2,))
  dnn = inputs
  for l_no in range(hidden_layers):
      dnn = Dense(2, activation='tanh', name = "layer_{}".format(l_no))(dnn)
  outputs = Dense(2, activation='softmax', name = "layer_out")(dnn)
  
  model = Model(inputs=inputs, outputs=outputs)
  
  model.compile(optimizer='adam', loss='categorical_crossentropy', metrics='accuracy'])
  
  The MLP never reached 100% accuracy. I tried to visualize how the MLP would classify the points of the 2D space with Black and White.
  limit = 4
  step = 0.2
  grid = []
  x = -limit
  while x <= limit:
      y = -limit
      while y <= limit:
          grid.append([x, y])
          y += step
      x += step
  grid = np.array(grid)
  
  prediction = model.predict(grid)
  
  This is the final image I get:
  xs = []
  ys = []
  cs = []
  for point in grid:
      xs.append(point[0])
      ys.append(point[1])
  for pred in prediction:
      cs.append(pred[0])
  
  plt.scatter(xs, ys, c = cs, s=70, cmap = 'gray')
  plt.scatter(classA_x, classA_y, c = 'r', s= 50)
  plt.scatter(classB_x, classB_y, c = 'b', s= 50)
  plt.show()
  
  
  At first, I was expecting that the MLP could classify 2 classes on a 2D space with 2 Neurons at each hidden layer, and I was expecting to see a white circle encapsulating the red points and the rest be a black space. Is there a (mathematical) reason, why the MLP fails to create a close shape, rather it seems to go from infinity to infinity on a 2d space ?? (Notice: If I use 3 neurons at each hidden layer, the MLP succeeds quite fast).
  What I mean by a closed shape, take a look at the second image which was generated by using 3 neurons at each layer:
  for l_no in range(hidden_layers):
      dnn = Dense(3, activation='tanh', name = "layer_{}".format(l_no))(dnn)
  
   
  [According to Marked Answer] 
  from keras import backend as K
  def x_squared(x):
      x = K.abs(x) * K.abs(x)
      return x
  hidden_layers = 3
  inputs = Input(shape=(2,))
  dnn = inputs
  for l_no in range(hidden_layers):
      dnn = Dense(2, activation=x_squared, name = "layer_{}".format(l_no))(dnn)
  outputs = Dense(2, activation='softsign', name = "layer_out")(dnn)
  model.compile(optimizer='adam',
                loss='mean_squared_error',
                metrics=['accuracy'])
  
  I get:
  
  **Tags:** machine-learning, keras, multilayer-perceptrons, linear-algebra
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-22. Score: 3 | Accepted answer
  
  In neural networks, the family of functions and the shapes that they can make for decision surfaces is determined by the activation function you use (in your case, `tanh` or hyperbolic tangent).
  
  Assuming at least one hidden layer, then the [universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem) applies. How closely you can approximate any given function is limited by the number of neurons, and not strongly by the choice of activation function. However, the choice of activation function is still relevant to how good an approximation is. When you get to low numbers of neurons in a hidden layer, then the approximations are more strongly tied to the nature of the activation function.
  
  With one neuron in a hidden layer, you can only approximate some affine transformation of the activation function. Other low numbers, such as 2, 3 etc, will still show strong tendencies for certain families of shapes. This is very similar conceptually to using limited number of frequencies in a Fourier transform - if you limit yourself to only $a\_1 \text{sin}(x) + a\_2 \text{sin}(2x)$ to approximate a function, then you will definitely notice the sinusoidal building blocks in any output.
  
  I suspect that if you changed the activation function in the first hidden layer to $f(x) = x^2$ then you could get a good result with two neurons per layer. If you then took that network and tried to train it on a simple linear split, it would fail, always producing some curved closed surface that covered the training examples as best that it could - kind of the opposite problem as you are seeing with your circular pattern fitted to NN with tanh activations throughout.
  
  One interesting thing about using $f(x) = x^2$ is that this is a deliberate choice (given knowledge about how you constructed the example) to map your input space to a new space where examples can be linearly separated. In fact this appears to be what layers in multi-layer NNs learn - each layer incrementally and progressively maps its input space to a new space where examples can be better separated in a linear fashion.
  
  > **Comments:**
  > **username_0**: You sufficiently covered my question, but I will wait a little bit until I mark this answer as the correct one. I will try to use x^2  
  > **username_1**: @entropyfeverone: Thank you. In general, $x^2$ makes a very bad activation function, it is only useful here because you already know the correct distribution can be modelled using it. The more usual thing to do in practice is to have sufficiently large layers.  
  > **username_0**: It worked :) Thank you. You can check the result in my question.  
  
  ---
  
  