Answers.Body: <p>I think this part is the art/experience part. The only certain thing is the deeper you go, the higher is the level of abstraction, and you don't always need your machine to philosophize.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-13T14:23:27.673Z
Answers.LastActivityDate: 2022-07-13T14:23:27.673Z
Answers.UserDisplayName: asdfasdfsdf
Answers.Body: <p>Designing neural network architectures from scratch for harder tasks is work usually performed by entire research groups (whether academic or business). There are, however, some things to keep in mind:</p>
  <ul>
  <li><p>Deeper networks have more abstraction, but also higher complexity. This means they can learn more complicated relations between input and output, but can also maximally encode the noise in the dataset. You may know this as the bias/variance trade-off, but is more often called over-fitting.</p>
  </li>
  <li><p>The type of data that goes into a network and the intuitive aspects of the data help decide what type of layers to use. Linear layers are your baseline. Convolutional layers when location is important (whether 1D, 2D or 3D). Attention layers for context, such as in language models. Recurrent connections for time-series data, such as speech patterns or observations over time.</p>
  </li>
  <li><p>Determining best practices on your own is an impossible task. There is a reason research is based on previous work. You may gain intuition over time, but decisions like when to use additional operations such as pooling, batch normalization, or which hyper parameters to use, such as activation functions, optimizers and learning rate, are best borrowed from existing work. Try to find work on a similar task to what you want to do and read some papers, preferably, or blogs and tutorials if the former is too complex.</p>
  </li>
  </ul>
  <p>If you have a specific task in mind, by all means post it as a comment to this answer and I can help you find similar work and/or suggest some intuitive starting point from my own experience.</p>
  <p>In response to your second question: You do not need to reduce the dimensionality of your data from the first layers of your model. In fact, you can expand it considerably. In the end you need to boil it down to the format of your output. For example classification using a CNN will first create feature maps of your data, which will increase dimensionality (e.g. 256x256x3 image to 128x128x64 feature maps), then it will flatten the features (128x128x64 feature maps to 1048576 hidden nodes) and finally it will classify (1048576 hidden nodes to 1000 output classes).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-14T13:08:42.843Z
Answers.LastActivityDate: 2022-07-14T13:13:43.637Z
Answers.LastEditDate: 2022-07-14T13:13:43.637Z
Answers.UserDisplayName: Kroshtan
Body: <p>I have become more familiar with libraries such as tensorflow for a while now, and have become interested in utilizing neural networks for solving specific problems. The big question I have is, what are some principles that you have to take into account for designing your neural networks architecture?</p>
  <p>Some other questions I have are:</p>
  <ul>
  <li>Do I want my network to slowly reduce the dimensionality of data (so
  it picks out important features), the deeper it goes? What happens
  when the output (lets say its one hot encoded, so the no. classes is
  in the 1000s while your e.g text is only of length 30) is a lot
  bigger than the input?</li>
  <li>If so, then what do I do when I have to process a single class? Do I just add layers which expand off that 1 input (isn't that wasting resources?)</li>
  </ul>
  <p>What resources do you recommend I should look into?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-14T13:14:41.177Z
Comments.Text: I have provided a pretty lengthy answer, but ideally you would post a single question at a time. Additionally, your third question (second "other" question) is unclear. What do you mean with single class? A single output value?
Comments.UserDisplayName: Kroshtan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-15T01:10:55.043Z
Comments.Text: @Kroshtan, sorry about that, I just did not want the question to be flagged as vague, by providing some specific questions to answer. What I meant is that there is a single input value, e.g username_id.
Comments.UserDisplayName: jr123456jr987654321
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-15T16:28:11.057Z
Comments.Text: A sample with a single value input and a single value output is a lookup table, there is nothing to learn regardless of expanding dimensions. If the input and output is numeric, you could technically learn a pattern such as `y=x^2`, but that's it.
Comments.UserDisplayName: Kroshtan
Tags: <neural-networks><machine-learning><resource-request>
Title: Principles of designing a neural network
UserDisplayName: jr123456jr987654321
ThreadText: # Principles of designing a neural network
  
  (Asked by: username_0 on 2022-07-13)
  
  I have become more familiar with libraries such as tensorflow for a while now, and have become interested in utilizing neural networks for solving specific problems. The big question I have is, what are some principles that you have to take into account for designing your neural networks architecture?
  
  Some other questions I have are:
  
  * Do I want my network to slowly reduce the dimensionality of data (so
    it picks out important features), the deeper it goes? What happens
    when the output (lets say its one hot encoded, so the no. classes is
    in the 1000s while your e.g text is only of length 30) is a lot
    bigger than the input?
  * If so, then what do I do when I have to process a single class? Do I just add layers which expand off that 1 input (isn't that wasting resources?)
  
  What resources do you recommend I should look into?
  
  **Tags:** neural-networks, machine-learning, resource-request
  
  > **Comments:**
  > **username_1**: I have provided a pretty lengthy answer, but ideally you would post a single question at a time. Additionally, your third question (second "other" question) is unclear. What do you mean with single class? A single output value?  
  > **username_0**: @username_1, sorry about that, I just did not want the question to be flagged as vague, by providing some specific questions to answer. What I meant is that there is a single input value, e.g username_id.  
  > **username_1**: A sample with a single value input and a single value output is a lookup table, there is nothing to learn regardless of expanding dimensions. If the input and output is numeric, you could technically learn a pattern such as `y=x^2`, but that's it.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-07-14. Score: 2 | Accepted answer
  
  Designing neural network architectures from scratch for harder tasks is work usually performed by entire research groups (whether academic or business). There are, however, some things to keep in mind:
  
  * Deeper networks have more abstraction, but also higher complexity. This means they can learn more complicated relations between input and output, but can also maximally encode the noise in the dataset. You may know this as the bias/variance trade-off, but is more often called over-fitting.
  * The type of data that goes into a network and the intuitive aspects of the data help decide what type of layers to use. Linear layers are your baseline. Convolutional layers when location is important (whether 1D, 2D or 3D). Attention layers for context, such as in language models. Recurrent connections for time-series data, such as speech patterns or observations over time.
  * Determining best practices on your own is an impossible task. There is a reason research is based on previous work. You may gain intuition over time, but decisions like when to use additional operations such as pooling, batch normalization, or which hyper parameters to use, such as activation functions, optimizers and learning rate, are best borrowed from existing work. Try to find work on a similar task to what you want to do and read some papers, preferably, or blogs and tutorials if the former is too complex.
  
  If you have a specific task in mind, by all means post it as a comment to this answer and I can help you find similar work and/or suggest some intuitive starting point from my own experience.
  
  In response to your second question: You do not need to reduce the dimensionality of your data from the first layers of your model. In fact, you can expand it considerably. In the end you need to boil it down to the format of your output. For example classification using a CNN will first create feature maps of your data, which will increase dimensionality (e.g. 256x256x3 image to 128x128x64 feature maps), then it will flatten the features (128x128x64 feature maps to 1048576 hidden nodes) and finally it will classify (1048576 hidden nodes to 1000 output classes).
  
  
  ---
  
  ### Answer by username_2 on 2022-07-13. Score: 1
  
  I think this part is the art/experience part. The only certain thing is the deeper you go, the higher is the level of abstraction, and you don't always need your machine to philosophize.
  
  
  ---
  
  