Answers.Body: <p>In a practical point of view, a &quot;layer&quot; in a neural network is a step in computation which only depends on its preceding layers' outputs. Note that layer <span class="math-container">$n$</span> might take inputs from any previous layers, not only from <span class="math-container">$n-1$</span>. Those are called skip connections. But since you are an expert, I assume you know all this already.</p>
  <p>Usually NNs are defined as a stack of layers, which define a DAG. But given a DAG, there are several valid layer-by-layer interpretations of its structure. But in general, I don't think it is important to know whether a given node is from layer <span class="math-container">$i$</span> or <span class="math-container">$j$</span>. May I ask, what is the motivation behind this question?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-10T17:30:16.633Z
Answers.Text: Thanks for the answer. As you stated, I am aware of all the facts you mentioned.
Answers.UserDisplayName: neoglez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-11T10:29:49.233Z
Answers.Text: @neoglez Ok, but then it's hard to believe that you're an expert and you are still confused about the concept. If this answers your question, [you should accept it](https://ai.stackexchange.com/help/someone-answers). If it doesn't then you should ask for clarifications. What's still confusing?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-12T11:22:02.257Z
Answers.Text: @nbro Please note, that the discussion ist not about whether I am expert or not. If you cerefully read the information you linked to, you can see that an answer can be accepted, when the answer satisfies the question, which ist unfortunately not the case.
Answers.UserDisplayName: neoglez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-13T09:41:28.853Z
Answers.Text: @nbro well, we can be experts in any field - *relatively speaking*. But usually an expert don't call himself one, in an believe-it-or-not tone. Beside, we should be careful when spelling the word "cerefully".
Answers.UserDisplayName: lpounng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-10T17:09:01.473Z
Answers.LastActivityDate: 2023-02-10T17:09:01.473Z
Answers.UserDisplayName: NikoNyrh
Body: <p>You may not believe it, but I am an ANN expert. Perhaps, for that reason, I am unable to grasp completely what the layers are in a Deep Forward Artificial Neural Network (DFANN).</p>
  <p>According to the <a href="https://www.deeplearningbook.org/contents/mlp.html" rel="nofollow noreferrer">Deep Learning &quot;bible&quot;, p. 164</a> &quot;the model is associated with a&quot; directed acyclic graph (DAG) &quot;describing how the functions are composed together&quot;. Then it states that one of these functions <span class="math-container">$f^{(i)}$</span> is a layer. We know that an ANN is composed of neurons/units (the biological perspective) and a DAG is composed of nodes/vertices and edges/links. A trivial mapping can be established between (neural) units and nodes, and (neural) connections and edges (see also <a href="https://www.deeplearningbook.org/contents/mlp.html" rel="nofollow noreferrer">Fig. 6.2 in p. 170</a>). However, defining the layers in a DAG is not that trivial, I would say.</p>
  <p>In a quick internet search, I came across about the concept of <a href="https://en.wikipedia.org/wiki/Graph_drawing" rel="nofollow noreferrer">graph drawing</a> and <a href="https://link.springer.com/content/pdf/10.1007/3-540-45848-4_2.pdf" rel="nofollow noreferrer">layering algorithms</a>. This concept is never mentioned in the ANN literature. If I understood well, formally, we would need such an algorithm to be able to define the layers in the DAG.</p>
  <p>A more intuitive description is given by <a href="https://aima.cs.berkeley.edu/" rel="nofollow noreferrer">AIMA</a>: DFANN &quot;are usually arranged in layers, such that each unit receives input only from units in the immediately preceding layer&quot;. That sounds indeed like a layering algorithm.</p>
  <p>So, formally, what are the layers in an Artificial Neural Network?</p>
  <p>As a side note, a practical argument is given by <a href="https://cs231n.github.io/neural-networks-1/" rel="nofollow noreferrer">https://cs231n.github.io/neural-networks-1/</a></p>
  <blockquote>
  <p>One of the primary reasons that Neural Networks are organized into layers is that this structure makes it very simple and efficient to evaluate Neural Networks using matrix vector operations.</p>
  </blockquote>
  <p>That's nice, but only from a practical perspective.</p>
  <p>An interesting remark has been made by Chillston (see comments, I quote):</p>
  <p>That's because an FFN is not a general DAG but a very regular kind of DAG, right.</p>
  <p>I also thought that could be the case. But <a href="http://people.sabanciuniv.edu/berrin/cs512/lectures/Book-Bishop-Neural%20Networks%20for%20Pattern%20Recognition.pdf" rel="nofollow noreferrer">Bishop, 1995, p. 116</a> contradicts us:</p>
  <blockquote>
  <p>More generally we can consider arbitrary network diagrams (not
  necessarily having a simple layered structure) since any network
  diagram can be converted into its corresponding mapping function. The
  only restriction is that the diagram must be feed-forward, so that it
  contains no feedback loops.</p>
  </blockquote>
  <p>As I see it, according to Bishop, any (general) DAG <em>is</em> indeed an ANN.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-10T20:39:50.620Z
Comments.Text: Can you specify your question or maybe describe what you mean exactly? From what I understand you are reasoning: An FFN is a DAG -> in a DAG the nodes cannot be trivially grouped into layers -> Neurons in an FFN cannot trivially be grouped into layers. Here, I would say that the way FFNs are constructed is very specific - i.e. one where this mapping from neurons to layers is trivial. That's because an FFN is not a general DAG but a very regular kind of DAG, right? If not what am I missing?
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-12T11:16:52.793Z
Comments.Text: You are indeed in the right path. The way you describe it, is exactly what mean. I will consider to complete the question with your remarks.
Comments.UserDisplayName: neoglez
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-15T18:26:55.107Z
Comments.Text: One thing that evades me a bit still the underlying reason for the question. Lets say you have a case of an ANN where a mapping from individual neurons to layers is not trivial. (When you think about it, maybe even the transformer is such a case because of the interactions of encoder and decoder and residual connections?). I cannot yet see why thinking about this neuron-to-layer mapping is useful. What would such a mapping be useful for, and, what conclusions could you draw if this mapping is non-trivial? I hope its clear what I mean.
Comments.UserDisplayName: Chillston
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-16T17:21:15.093Z
Comments.Text: Well, regarding Q. No. 1, one could ask the same for every aspect of research and science, especially in graph theory and topology.
  
  The 2. Q. is a little more perturbing because it uses the argument of utility (Utilitarianism?) to establish the importance of the matter.
  
  Philosophical discussions apart, my motivation in asking this question is rather banal. I am writing my dissertation and I realized that I was unable to connect the arguments fluently. The details would need several pages, so I will stop here.
  I almost found the answer, but I have to double-check it before I post it.
Comments.UserDisplayName: neoglez
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-16T18:52:22.323Z
Comments.Text: Sure, the main reason why I was asking that is that a questions sounds a bit confusing if such details aren't clarified (i.e. that this is an explorative question with yet unknown utility). I wish you success with the diss and have my fingers crossed for a positive outcome of this investigation. Also I am actually curious about your argument and in case you find some interesting properties regarding the layering question, maybe you can drop a link or summary ;)
Comments.UserDisplayName: Chillston
Tags: <neural-networks><deep-learning><definitions><deep-neural-networks><layers>
Title: Formally, what are the layers in an Artificial Neural Network?
UserDisplayName: neoglez
ThreadText: # Formally, what are the layers in an Artificial Neural Network?
  
  (Asked by: username_0 on 2023-02-10)
  
  You may not believe it, but I am an ANN expert. Perhaps, for that reason, I am unable to grasp completely what the layers are in a Deep Forward Artificial Neural Network (DFANN).
  
  According to the [Deep Learning "bible", p. 164](https://www.deeplearningbook.org/contents/mlp.html) "the model is associated with a" directed acyclic graph (DAG) "describing how the functions are composed together". Then it states that one of these functions $f^{(i)}$ is a layer. We know that an ANN is composed of neurons/units (the biological perspective) and a DAG is composed of nodes/vertices and edges/links. A trivial mapping can be established between (neural) units and nodes, and (neural) connections and edges (see also [Fig. 6.2 in p. 170](https://www.deeplearningbook.org/contents/mlp.html)). However, defining the layers in a DAG is not that trivial, I would say.
  
  In a quick internet search, I came across about the concept of [graph drawing](https://en.wikipedia.org/wiki/Graph_drawing) and [layering algorithms](https://link.springer.com/content/pdf/10.1007/3-540-45848-4_2.pdf). This concept is never mentioned in the ANN literature. If I understood well, formally, we would need such an algorithm to be able to define the layers in the DAG.
  
  A more intuitive description is given by [AIMA](https://aima.cs.berkeley.edu/): DFANN "are usually arranged in layers, such that each unit receives input only from units in the immediately preceding layer". That sounds indeed like a layering algorithm.
  
  So, formally, what are the layers in an Artificial Neural Network?
  
  As a side note, a practical argument is given by <https://cs231n.github.io/neural-networks-1/>
  
  > One of the primary reasons that Neural Networks are organized into layers is that this structure makes it very simple and efficient to evaluate Neural Networks using matrix vector operations.
  
  That's nice, but only from a practical perspective.
  
  An interesting remark has been made by username_1 (see comments, I quote):
  
  That's because an FFN is not a general DAG but a very regular kind of DAG, right.
  
  I also thought that could be the case. But [Bishop, 1995, p. 116](http://people.sabanciuniv.edu/berrin/cs512/lectures/Book-Bishop-Neural%20Networks%20for%20Pattern%20Recognition.pdf) contradicts us:
  
  > More generally we can consider arbitrary network diagrams (not
  > necessarily having a simple layered structure) since any network
  > diagram can be converted into its corresponding mapping function. The
  > only restriction is that the diagram must be feed-forward, so that it
  > contains no feedback loops.
  
  As I see it, according to Bishop, any (general) DAG *is* indeed an ANN.
  
  **Tags:** neural-networks, deep-learning, definitions, deep-neural-networks, layers
  
  > **Comments:**
  > **username_1**: Can you specify your question or maybe describe what you mean exactly? From what I understand you are reasoning: An FFN is a DAG -> in a DAG the nodes cannot be trivially grouped into layers -> Neurons in an FFN cannot trivially be grouped into layers. Here, I would say that the way FFNs are constructed is very specific - i.e. one where this mapping from neurons to layers is trivial. That's because an FFN is not a general DAG but a very regular kind of DAG, right? If not what am I missing?  
  > **username_0**: You are indeed in the right path. The way you describe it, is exactly what mean. I will consider to complete the question with your remarks.  
  > **username_1**: One thing that evades me a bit still the underlying reason for the question. Lets say you have a case of an ANN where a mapping from individual neurons to layers is not trivial. (When you think about it, maybe even the transformer is such a case because of the interactions of encoder and decoder and residual connections?). I cannot yet see why thinking about this neuron-to-layer mapping is useful. What would such a mapping be useful for, and, what conclusions could you draw if this mapping is non-trivial? I hope its clear what I mean.  
  > **username_0**: Well, regarding Q. No. 1, one could ask the same for every aspect of research and science, especially in graph theory and topology.  The 2. Q. is a little more perturbing because it uses the argument of utility (Utilitarianism?) to establish the importance of the matter.  Philosophical discussions apart, my motivation in asking this question is rather banal. I am writing my dissertation and I realized that I was unable to connect the arguments fluently. The details would need several pages, so I will stop here. I almost found the answer, but I have to double-check it before I post it.  
  > **username_1**: Sure, the main reason why I was asking that is that a questions sounds a bit confusing if such details aren't clarified (i.e. that this is an explorative question with yet unknown utility). I wish you success with the diss and have my fingers crossed for a positive outcome of this investigation. Also I am actually curious about your argument and in case you find some interesting properties regarding the layering question, maybe you can drop a link or summary ;)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-02-10. Score: 1
  
  In a practical point of view, a "layer" in a neural network is a step in computation which only depends on its preceding layers' outputs. Note that layer $n$ might take inputs from any previous layers, not only from $n-1$. Those are called skip connections. But since you are an expert, I assume you know all this already.
  
  Usually NNs are defined as a stack of layers, which define a DAG. But given a DAG, there are several valid layer-by-layer interpretations of its structure. But in general, I don't think it is important to know whether a given node is from layer $i$ or $j$. May I ask, what is the motivation behind this question?
  
  > **Comments:**
  > **username_0**: Thanks for the answer. As you stated, I am aware of all the facts you mentioned.  
  > **username_3**: @username_0 Ok, but then it's hard to believe that you're an expert and you are still confused about the concept. If this answers your question, [you should accept it](https://ai.stackexchange.com/help/someone-answers). If it doesn't then you should ask for clarifications. What's still confusing?  
  > **username_0**: @username_3 Please note, that the discussion ist not about whether I am expert or not. If you cerefully read the information you linked to, you can see that an answer can be accepted, when the answer satisfies the question, which ist unfortunately not the case.  
  > **username_4**: @username_3 well, we can be experts in any field - *relatively speaking*. But usually an expert don't call himself one, in an believe-it-or-not tone. Beside, we should be careful when spelling the word "cerefully".  
  
  ---
  
  