Answers.Body: <p>So if I understand correctly, you're proposing to use a neutral net with <span class="math-container">$N$</span> input units (let's say data is in <span class="math-container">$\mathbf{R}^N$</span>), 1 hidden unit, and whatever the necessary output needs to be.</p>
  <p>Let's say we try to do this. Then each unit of the output layer is responsible for computing its output based on a single scalar input. So it's like as if you're saying the problem can be compressed to another problem that is 1-dimensional. Now let's say you're doing regression. By modeling your NN this way, you're saying there's a function that (approximately) maps a scalar (one dimensional object) to the target output. Suppose your output layer is a linear layer (which is usually the case in regression), this is saying that the dimension of the output space is at most 1. This is almost never the case, especially with complex data.</p>
  <p>Now suppose you had several hidden units (call this number <span class="math-container">$M$</span>) and the weights are initialized properly. Then, it's likely that your hidden unit will have dimension larger than one, perhaps even <span class="math-container">$M$</span>, which allows for a richer output space as <span class="math-container">$M$</span> increases.</p>
  <p>You can make similar arguments for the classification setting, but the output nonlinearity just makes it a little less clear.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-25T13:44:47.710Z
Answers.Text: Note that this does not answer the question "_why_... is not enough?", but it's just an attempt to interpret the situation.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-25T15:54:58.113Z
Answers.Text: Sorry, about that, just added some elaboration.
Answers.UserDisplayName: harwiltz
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-25T13:32:58.943Z
Answers.LastActivityDate: 2020-09-25T15:54:34.570Z
Answers.LastEditDate: 2020-09-25T15:54:34.570Z
Answers.UserDisplayName: harwiltz
Body: <p>In a deep connected network, when every unit gets all the input features(X) so it has one parameter for every feature and every unit tweaks its parameters for loss optimization. What if we use only one unit and that one unit will have all the parameters which it can tweak for loss optimization. Is there a reason or benefit of using multiple units in every layer except the output layer?</p>
  
Tags: <neural-networks><deep-learning><artificial-neuron><hyper-parameters>
Title: Why one unit in the layers of neural network is not enough?
UserDisplayName: Hitesh Somani
ThreadText: # Why one unit in the layers of neural network is not enough?
  
  (Asked by: username_0 on 2020-09-24)
  
  In a deep connected network, when every unit gets all the input features(X) so it has one parameter for every feature and every unit tweaks its parameters for loss optimization. What if we use only one unit and that one unit will have all the parameters which it can tweak for loss optimization. Is there a reason or benefit of using multiple units in every layer except the output layer?
  
  **Tags:** neural-networks, deep-learning, artificial-neuron, hyper-parameters
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-25. Score: 1
  
  So if I understand correctly, you're proposing to use a neutral net with $N$ input units (let's say data is in $\mathbf{R}^N$), 1 hidden unit, and whatever the necessary output needs to be.
  
  Let's say we try to do this. Then each unit of the output layer is responsible for computing its output based on a single scalar input. So it's like as if you're saying the problem can be compressed to another problem that is 1-dimensional. Now let's say you're doing regression. By modeling your NN this way, you're saying there's a function that (approximately) maps a scalar (one dimensional object) to the target output. Suppose your output layer is a linear layer (which is usually the case in regression), this is saying that the dimension of the output space is at most 1. This is almost never the case, especially with complex data.
  
  Now suppose you had several hidden units (call this number $M$) and the weights are initialized properly. Then, it's likely that your hidden unit will have dimension larger than one, perhaps even $M$, which allows for a richer output space as $M$ increases.
  
  You can make similar arguments for the classification setting, but the output nonlinearity just makes it a little less clear.
  
  > **Comments:**
  > **username_2**: Note that this does not answer the question "_why_... is not enough?", but it's just an attempt to interpret the situation.  
  > **username_1**: Sorry, about that, just added some elaboration.  
  
  ---
  
  