Answers.Body: <p>Evidentially, one of the most important concepts in NN design are <strong>inductive biases and symmetry</strong>. A good way to approach the concept of symmetry is to ask: Given a datapoint <span class="math-container">$x \in \mathcal{D}$</span> from my Dataset <span class="math-container">$\mathcal{D}$</span>, how can I transform <span class="math-container">$x$</span> without changing it's meaning. Indictive biases, on the other hand, are essentially assumptions that you have about your data and that you embed into your model architecture.</p>
  <p>Here are some examples of what I mean:</p>
  <p><strong>Images:</strong> Images have a <a href="https://en.wikipedia.org/wiki/Translational_symmetry" rel="nofollow noreferrer">translational symmetry</a>, meaning the identity of an object doesn't change when you move it to another position in the image - a cat in the left corner of an image remains a cat even if you move it to the right corner.
  If you now want to classify images and where to take an MLP instead of a ConvNet, your model had to learn to identify a cat in every possible position of the image, because the inputs to your model change with the slightest translation. In contrast, ConvNets have a translational inductive bias, i.e. are translation equivariant. This means they have a build in robustness against translations that requires no learning. This is one of the reasons, they are so good with images. This property of ConvNets arises from applying the same weights (kernel) over the space of the image. Another <em>inductive bias</em> of ConvNets is locality: The full image is not so important - whats important is local patterns and groups of pixels.</p>
  <p><strong>Text:</strong>
  In AI research text was understood as sequential data. This is why people came up with RNNs like GRUs and LSTMs, thus implementing models with a sequential inductive bias. However, since transformers took over it has become clearer that the position of a word in a sentence is important but not primarily encoding the meaning. Instead a graph-like relation between words is more discriminative, which IMO, is why attention-based models are now the SOTA. How I understand this is that words have a specific permutation symmetry that depends on grammar and retains meaning if you permute words in a sentence in a specific way.</p>
  <p><strong>Point Clouds:</strong>
  A 3D scan (point cloud) remains the same under rotation and translation in 3D, i.e. point clouds are SE(3)-symmetric. In the paper [2], this is biased into a transformer model by a clever usage of self-attention.</p>
  <p><strong>Graphs:</strong> The identity of a graph doesn't change under permutation as long as the relations between nodes remain the same. Hence, graph neural networks are invariant to permutation.</p>
  <p>The list goes on like this and I put some references at the end for further reading - there is a lot of material regarding these topics and IMO it is super important. I'd recommend reading [1] which is very clearly written, [3] is very technical and I found it pretty hard to understand in detail, but it provides a nice overview of symmetries and provides an intuition of what constitutes groups and how this is related to equivariance and invariance in neural networks.</p>
  <hr />
  <p>In your specific use case, you could ask: What changes can I apply to the input without changing the expected prediction and is there some sort of hierarchy or any other bias that could be useful. I'm no protein expert, but I assume a reflection symmetry (reversing the input sequence). I'd also assume that sequence length is important but not very descriptive. There might be local motives that are more important (Like image size is not important but local groups of pixels are). This would direct you to a localized model such as 1D Convolutions. Providing the model with reflection equivariance is straight forward in the case of 1D convolutions: You can, for every learned kernel, add the reversed kernel to the convolution operations.</p>
  <p>This got pretty long by now, but I hope it is somewhat clear what I mean by all of this.</p>
  <hr />
  <ul>
  <li><a href="http://arxiv.org/abs/2104.13478" rel="nofollow noreferrer">[1] Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges</a></li>
  <li><a href="https://arxiv.org/abs/2006.10503" rel="nofollow noreferrer">[2] SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks</a></li>
  <li><a href="https://arxiv.org/abs/2104.09459" rel="nofollow noreferrer">[3] A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups</a></li>
  <li><a href="https://www.youtube.com/watch?v=YihnfamwA_s" rel="nofollow noreferrer">[4] Video: Nice talk of Max Welling on the topic of symmetry</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-24T10:54:27.673Z
Answers.LastActivityDate: 2022-11-24T10:54:27.673Z
Answers.UserDisplayName: Chillston
Answers.Body: <p>I'll try to answer, because even this is beyond the scope of my area I think it may help.</p>
  <blockquote>
  <p>How do I know what NN type would best serve our purpose? How do I know the number of layers we need in the model? How do I know if we need a feedback loop or not?</p>
  </blockquote>
  <p>This generally depends on the level of experience. There is not deterministic way how to decide and find an answer for these.</p>
  <blockquote>
  <p>How can I improve or redesign the NN model?</p>
  </blockquote>
  <p>By playing with it I suppose.</p>
  <p>I have not much experience in this area and zero in protein reconstruction tasks. But I think that the best way would be as in any other area of any work.</p>
  <ol>
  <li>I would to simplify the task and model based on gained knowledge.</li>
  <li>Use this network to solve couple more similar problems, prune it and compare pruned networks.</li>
  <li>I like the idea of Generative adversarial network.</li>
  <li>Design neural net to help to design your solution.</li>
  <li>Or evolution algorithm should to find some answers also.</li>
  </ol>
  <p>I think it is mainly about to get more knowledge about the solution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-23T19:54:37.383Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-23T16:52:27.153Z
Answers.LastActivityDate: 2022-11-23T16:52:27.153Z
Answers.UserDisplayName: dominikj111
Body: <p>We developed a neural network-based protein reconstruction tool to reconstruct the main chain from only CA atoms.</p>
  <ol>
  <li>we generated data from some selected PDBs from the RCSB website to train an NN model.</li>
  <li>we then use those data to train the NN model.</li>
  <li>we select some test PDBs, strip all atoms except CA atoms, and save them in files.</li>
  <li>we pass those CA-only PDBs through the NN model, obtain a reconstructed main chain, and save them in files.</li>
  <li>we compare original and reconstructed PDB files and calculate CRMSD values.</li>
  </ol>
  <p>The data set is large: a sample of 1398438 rows and 102 columns (data points). The model is Keras via a 4-layered MLP. There is no feedback loop or convolution applied.</p>
  <p>We obtained a CRMSD value of 0.3559, which is not satisfactory.</p>
  <hr />
  <p>How can I improve or redesign the NN model?</p>
  <p>How do I know what NN type would best serve our purpose? How do I know the number of layers we need in the model? How do I know if we need a feedback loop or not?</p>
  
Tags: <neural-networks>
Title: How do I know how to design my ANN so that it fits my specific problem?
UserDisplayName: user366312
ThreadText: # How do I know how to design my ANN so that it fits my specific problem?
  
  (Asked by: username_0 on 2022-11-19)
  
  We developed a neural network-based protein reconstruction tool to reconstruct the main chain from only CA atoms.
  
  1. we generated data from some selected PDBs from the RCSB website to train an NN model.
  2. we then use those data to train the NN model.
  3. we select some test PDBs, strip all atoms except CA atoms, and save them in files.
  4. we pass those CA-only PDBs through the NN model, obtain a reconstructed main chain, and save them in files.
  5. we compare original and reconstructed PDB files and calculate CRMSD values.
  
  The data set is large: a sample of 1398438 rows and 102 columns (data points). The model is Keras via a 4-layered MLP. There is no feedback loop or convolution applied.
  
  We obtained a CRMSD value of 0.3559, which is not satisfactory.
  
  ---
  
  How can I improve or redesign the NN model?
  
  How do I know what NN type would best serve our purpose? How do I know the number of layers we need in the model? How do I know if we need a feedback loop or not?
  
  **Tags:** neural-networks
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-11-24. Score: 2
  
  Evidentially, one of the most important concepts in NN design are **inductive biases and symmetry**. A good way to approach the concept of symmetry is to ask: Given a datapoint $x \in \mathcal{D}$ from my Dataset $\mathcal{D}$, how can I transform $x$ without changing it's meaning. Indictive biases, on the other hand, are essentially assumptions that you have about your data and that you embed into your model architecture.
  
  Here are some examples of what I mean:
  
  **Images:** Images have a [translational symmetry](https://en.wikipedia.org/wiki/Translational_symmetry), meaning the identity of an object doesn't change when you move it to another position in the image - a cat in the left corner of an image remains a cat even if you move it to the right corner.
  If you now want to classify images and where to take an MLP instead of a ConvNet, your model had to learn to identify a cat in every possible position of the image, because the inputs to your model change with the slightest translation. In contrast, ConvNets have a translational inductive bias, i.e. are translation equivariant. This means they have a build in robustness against translations that requires no learning. This is one of the reasons, they are so good with images. This property of ConvNets arises from applying the same weights (kernel) over the space of the image. Another *inductive bias* of ConvNets is locality: The full image is not so important - whats important is local patterns and groups of pixels.
  
  **Text:**
  In AI research text was understood as sequential data. This is why people came up with RNNs like GRUs and LSTMs, thus implementing models with a sequential inductive bias. However, since transformers took over it has become clearer that the position of a word in a sentence is important but not primarily encoding the meaning. Instead a graph-like relation between words is more discriminative, which IMO, is why attention-based models are now the SOTA. How I understand this is that words have a specific permutation symmetry that depends on grammar and retains meaning if you permute words in a sentence in a specific way.
  
  **Point Clouds:**
  A 3D scan (point cloud) remains the same under rotation and translation in 3D, i.e. point clouds are SE(3)-symmetric. In the paper [2], this is biased into a transformer model by a clever usage of self-attention.
  
  **Graphs:** The identity of a graph doesn't change under permutation as long as the relations between nodes remain the same. Hence, graph neural networks are invariant to permutation.
  
  The list goes on like this and I put some references at the end for further reading - there is a lot of material regarding these topics and IMO it is super important. I'd recommend reading [1] which is very clearly written, [3] is very technical and I found it pretty hard to understand in detail, but it provides a nice overview of symmetries and provides an intuition of what constitutes groups and how this is related to equivariance and invariance in neural networks.
  
  ---
  
  In your specific use case, you could ask: What changes can I apply to the input without changing the expected prediction and is there some sort of hierarchy or any other bias that could be useful. I'm no protein expert, but I assume a reflection symmetry (reversing the input sequence). I'd also assume that sequence length is important but not very descriptive. There might be local motives that are more important (Like image size is not important but local groups of pixels are). This would direct you to a localized model such as 1D Convolutions. Providing the model with reflection equivariance is straight forward in the case of 1D convolutions: You can, for every learned kernel, add the reversed kernel to the convolution operations.
  
  This got pretty long by now, but I hope it is somewhat clear what I mean by all of this.
  
  ---
  
  * [[1] Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](http://arxiv.org/abs/2104.13478)
  * [[2] SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://arxiv.org/abs/2006.10503)
  * [[3] A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups](https://arxiv.org/abs/2104.09459)
  * [[4] Video: Nice talk of Max Welling on the topic of symmetry](https://www.youtube.com/watch?v=YihnfamwA_s)
  
  
  ---
  
  ### Answer by username_2 on 2022-11-23. Score: 0
  
  I'll try to answer, because even this is beyond the scope of my area I think it may help.
  
  > How do I know what NN type would best serve our purpose? How do I know the number of layers we need in the model? How do I know if we need a feedback loop or not?
  
  This generally depends on the level of experience. There is not deterministic way how to decide and find an answer for these.
  
  > How can I improve or redesign the NN model?
  
  By playing with it I suppose.
  
  I have not much experience in this area and zero in protein reconstruction tasks. But I think that the best way would be as in any other area of any work.
  
  1. I would to simplify the task and model based on gained knowledge.
  2. Use this network to solve couple more similar problems, prune it and compare pruned networks.
  3. I like the idea of Generative adversarial network.
  4. Design neural net to help to design your solution.
  5. Or evolution algorithm should to find some answers also.
  
  I think it is mainly about to get more knowledge about the solution.
  
  > **Comments:**
  > **username_3**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  