Are the number of neurons in a hidden layer equal to the dimensions in which a hyperplane is being sought by the network? My question stems from this example and my basic understanding of neural networks so far. Forgive my ignorance if I am totally off.
So in this example, a neural network with 2 neurons cannot find a representation for this data that fully separates the two.

A third neuron must be added after which the problem can easily be solved by a neural network. So by adding the third neuron, the network can map the data into a three dimensional space and find a representation that can be linearly separated.

My question that arises from this: are the number of neurons in a hidden layer equal to the dimensional space in which a hyperplane is being sought by the network?

I would argue that your intuition is correct for this simple example.
There are two components to this simplicity:

*

*A separating hyper-plane is something that is only really meaningful in binary classification. For multi-class problems this intuition might still work. However, you would have decision boundaries that are combinations of multiple hyper-planes. However, for regression, there is typically no such thing as a separating hyper-plane or decision boundary.

*When you have a single hidden layer the activations this layer produces can be interpreted as a feature vector that will be used as input to a logistic regression (the final layer in your network). Therefore, this reasoning always applies to the activations for the last hidden layer, which is equivalent to the penultimate layer of your entire network. However, in deeper networks, the other hidden layers are in some latent space that would be hard to understand. Since they are (typically) not the input to some linear model (where we have some intuition for hyper-planes etc.), there will be no hyper-planes in this context. It is likely that some sort of non-linear(!) decision boundaries will emerge, but I am not sure how well these are understood at this point.

