Question: In a basic 2 layer neural net, there are (for example) 3 inputs that each go into each neuron. If the same inputs are going into each neuron and we are applying the same optimization algorithm of gradient descent as well as the same sigmoid activation function, wouldn't they all give you the same result rendering the extra neurons useless? 

What I see is that you would apply gradient descent to each of the randomly chosen weights and biases, and eventually, they would all reach the same value since all the other functions are kept constant. 

Can anyone please explain what I'm missing here? Thanks!

Answer: 
If all weights are initialized with the same values, all neurons in each layer give you the same outputs. This is the reason that the weights are initialized with random numbers.
Let say we have an input vector $\vec{X\_0}$ and two neurons $A$ and $B$:

$F(A) = \sigma(W\_{0A}\*\vec X\_0-\beta\_A)$ 

and 

$F(B) = \sigma(W\_{0B}\*\vec X\_0-\beta\_B)$ 

accordingly, where $\sigma$ is the activation function (let's say sigmoid) and $W$ is the weights matrix from layer 0 (input) to layer 1 (A,B).

The results of A and B would be different if the weight of A and B are different.

The gradient update won't necessary be the same:

* First, A and B may lead in further layers to different calculations, and therefore their effect on the loss function ($\mathcal L(\vec X)$) won't be the same. For example: $\mathcal L(\vec X) = w\_A\*A + w\_B\*B -Y $
* Moreover, even if the weight are different, the deriviation of $\frac{ \mathcal L(\vec X)}{dW}$ won't be the same for $W\_{0A}$ and $W\_{0B}$, and therefore the update will result in different values.