Answers.Body: <p>Yes, the order of hidden layers can impact how information is processed in a neural network, it will also affect how the neural network is trained</p>
  <p>I would recommend you use 1st option with a pattern of   Decreasing Layer Sizes (10→32→24→16→8→1)</p>
  <p>This structure starts with a wider layer, allowing the network to capture more features or information from the input. As you move deeper, the number of neurons decreases, progressively reducing the dimensionality of the learned representations.</p>
  <p>This is the most common architecture used in various neural networks as larger starting layers can capture more complex problems and the gradual reduction helps distill that information toward the final prediction.</p>
  <p>I would recommend it for the regression task since it starts by expanding the input features and then condenses the information toward the single output. It allows for a gradual reduction in the feature space, making it easier for the network to learn hierarchical features.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-23T04:23:00.060Z
Answers.LastActivityDate: 2024-10-23T04:23:00.060Z
Answers.UserDisplayName: Keval Pandya
Answers.Body: <p>There is technically no general rule as to the order of layer widths.  If the model cannot draw a low loss linear line in the original dimensionality, transforming it to a higher dimension may help.</p>
  <p>Once one is in a higher dimension it often makes sense to only go down (not #2’s rollercoaster).  But it’s also possible a transformation to a lower dimension simplifies the data and then a transformation to a higher dimension makes it linear.</p>
  <p>All of that being said, I would typically start at one, proceed to 3 if something in the data really pointed to that transformation helping and then might end up at 2 if I’m just throwing stuff at the wall.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-23T02:35:53.790Z
Answers.Text: A prominent example of a "rollercoaster" architecture is the transformer model, which uses feedforward layers that project to a larger dimension and then back down to a smaller dimension. This is mentioned in the "Attention is all you need" paper.
Answers.UserDisplayName: Sycorax
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-23T01:57:43.537Z
Answers.LastActivityDate: 2024-10-23T01:57:43.537Z
Answers.UserDisplayName: foreverska
Body: <p>Say I have a Neural Network for a regression task that takes in 10 inputs and has 1 output, with 4 hidden layers of sizes 32, 24, 16, 8. Which order of hidden layers makes more sense?</p>
  <p><span class="math-container">$1. 10 \rightarrow 32 \rightarrow 24 \rightarrow 16 \rightarrow 8 \rightarrow 1$</span></p>
  <p><span class="math-container">$2. 10 \rightarrow 32 \rightarrow 16 \rightarrow 24 \rightarrow 8 \rightarrow 1$</span></p>
  <p><span class="math-container">$3. 10 \rightarrow 8 \rightarrow 16 \rightarrow 24 \rightarrow 32 \rightarrow 1$</span></p>
  <p>I feel like it's not going to be the increasing order, because from 10 neurons to 8 neurons we lose some information about the input whereas from 10 to 32 neurons we can preserve different information about the input. But I'm not sure about the second one, whether if it's worse than the decreasing order or not.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-10-27T22:04:44.387Z
Comments.Text: This apparently simple question is not so simple to answer, as far as I know, and it's also interesting.
Comments.UserDisplayName: nbro
Tags: <neural-networks><hidden-layers>
Title: How to determine the order of hidden layers?
UserDisplayName: hansoko
ThreadText: # How to determine the order of hidden layers?
  
  (Asked by: username_0 on 2024-10-22)
  
  Say I have a Neural Network for a regression task that takes in 10 inputs and has 1 output, with 4 hidden layers of sizes 32, 24, 16, 8. Which order of hidden layers makes more sense?
  
  $1. 10 \rightarrow 32 \rightarrow 24 \rightarrow 16 \rightarrow 8 \rightarrow 1$
  
  $2. 10 \rightarrow 32 \rightarrow 16 \rightarrow 24 \rightarrow 8 \rightarrow 1$
  
  $3. 10 \rightarrow 8 \rightarrow 16 \rightarrow 24 \rightarrow 32 \rightarrow 1$
  
  I feel like it's not going to be the increasing order, because from 10 neurons to 8 neurons we lose some information about the input whereas from 10 to 32 neurons we can preserve different information about the input. But I'm not sure about the second one, whether if it's worse than the decreasing order or not.
  
  **Tags:** neural-networks, hidden-layers
  > **Comments:**
  > **username_1**: This apparently simple question is not so simple to answer, as far as I know, and it's also interesting.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-10-23. Score: 1
  
  Yes, the order of hidden layers can impact how information is processed in a neural network, it will also affect how the neural network is trained
  
  I would recommend you use 1st option with a pattern of Decreasing Layer Sizes (10→32→24→16→8→1)
  
  This structure starts with a wider layer, allowing the network to capture more features or information from the input. As you move deeper, the number of neurons decreases, progressively reducing the dimensionality of the learned representations.
  
  This is the most common architecture used in various neural networks as larger starting layers can capture more complex problems and the gradual reduction helps distill that information toward the final prediction.
  
  I would recommend it for the regression task since it starts by expanding the input features and then condenses the information toward the single output. It allows for a gradual reduction in the feature space, making it easier for the network to learn hierarchical features.
  
  
  ---
  
  ### Answer by username_3 on 2024-10-23. Score: 1
  
  There is technically no general rule as to the order of layer widths. If the model cannot draw a low loss linear line in the original dimensionality, transforming it to a higher dimension may help.
  
  Once one is in a higher dimension it often makes sense to only go down (not #2’s rollercoaster). But it’s also possible a transformation to a lower dimension simplifies the data and then a transformation to a higher dimension makes it linear.
  
  All of that being said, I would typically start at one, proceed to 3 if something in the data really pointed to that transformation helping and then might end up at 2 if I’m just throwing stuff at the wall.
  
  > **Comments:**
  > **username_4**: A prominent example of a "rollercoaster" architecture is the transformer model, which uses feedforward layers that project to a larger dimension and then back down to a smaller dimension. This is mentioned in the "Attention is all you need" paper.  
  
  ---
  
  