Answers.Body: <p>In a FFN, a layer is a set of functions <span class="math-container">$f_1, f_2, \dots, f_n$</span> like this <span class="math-container">$f_j = \sigma(x_1 * w_{j1} + \dots + x_n * w_{jm})$</span> (sometimes called neurons), where <span class="math-container">$x_i$</span> and <span class="math-container">$w_i$</span> are respectively the inputs and parameters, and <span class="math-container">$\sigma$</span> is the activation function. So, <span class="math-container">$m$</span> is the number of inputs each function receives, which is equal to the number of parameters, while <span class="math-container">$n$</span> is the number of functions in the layer. You might have noticed that I wrote <span class="math-container">$w_{j1}$</span> instead of <span class="math-container">$w_1$</span> - this is intentional - each function has its own parameters.</p>
  <p>Now, can you write the computation of a layer, set of functions, as a matrix multiplication? Hint: yes</p>
  <p>Last, notice that <span class="math-container">$f_j$</span> becomes <span class="math-container">$x_j$</span> for the successive layer. So you can stack layers. Neural networks are just composite parametrized functions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-10T02:17:32.983Z
Answers.LastActivityDate: 2023-11-10T02:17:32.983Z
Answers.UserDisplayName: nbro
Answers.Body: <p>For your example, <span class="math-container">$W_2$</span>, <span class="math-container">$b_2$</span>, and whatever activation function you use would collectively be considered the last layer. But for a more complicated architecture, such as a stack of LSTMs, the set of (matrix-vector multiplications + activations) that are necessary for a single LSTM gate might be considered a &quot;layer&quot;.</p>
  <p>There's no formal definition of &quot;layer&quot;. We call a group of calculations a &quot;layer&quot; whenever thinking about them as a single unit make it easier to reason about how the network as a whole works.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-20T21:38:35.197Z
Answers.LastActivityDate: 2023-11-20T21:38:35.197Z
Answers.UserDisplayName: Ray
Body: <p>I'm working my way through how LLMs work and I understand how things work but it's not clear to me exactly what is semantically defined as a &quot;layer&quot;.</p>
  <p>Using the following FFN as an example:</p>
  <ol>
  <li>The weight matrices will be <span class="math-container">$W_1$</span> of size <span class="math-container">$[4 \times 6]$</span> and <span class="math-container">$W_2$</span> of size <span class="math-container">$[6 \times 4]$</span>.</li>
  <li>The bias vectors will be <span class="math-container">$b_1$</span> of size <span class="math-container">$[6]$</span> and <span class="math-container">$b_2$</span> of size <span class="math-container">$[4]$</span>.</li>
  </ol>
  <p><span class="math-container">$$W_1 = \begin{pmatrix} 0.1 &amp; 0.2 &amp; 0.3 &amp; 0.4 &amp; 0.5 &amp; 0.6 \\\ 0.7 &amp; 0.8 &amp; 0.9 &amp; 1.0 &amp; 1.1 &amp; 1.2 \\\ 1.3 &amp; 1.4 &amp; 1.5 &amp; 1.6 &amp; 1.7 &amp; 1.8 \\\ 1.9 &amp; 2.0 &amp; 2.1 &amp; 2.2 &amp; 2.3 &amp; 2.4 \end{pmatrix}$$</span></p>
  <p><span class="math-container">$$b_1 = \begin{pmatrix} 0.01 \\\ 0.02 \\\ 0.03 \\\ 0.04 \\\ 0.05 \\\ 0.06 \end{pmatrix}$$</span></p>
  <p><span class="math-container">$$W_2 = \begin{pmatrix} 0.1 &amp; 0.7 &amp; 1.3 &amp; 1.9 \\\ 0.2 &amp; 0.8 &amp; 1.4 &amp; 2.0 \\\ 0.3 &amp; 0.9 &amp; 1.5 &amp; 2.1 \\\ 0.4 &amp; 1.0 &amp; 1.6 &amp; 2.2 \\\ 0.5 &amp; 1.1 &amp; 1.7 &amp; 2.3 \\\ 0.6 &amp; 1.2 &amp; 1.8 &amp; 2.4 \end{pmatrix}$$</span></p>
  <p><span class="math-container">$$b_2 = \begin{pmatrix} 0.01 \\\ 0.02 \\\ 0.03 \\\ 0.04 \end{pmatrix}$$</span></p>
  <p>and just in case it matters, let's say that the input from layer normalization is:</p>
  <p><span class="math-container">$$ \text{LN}(x_i) = \begin{pmatrix} -1.03927142 &amp; 0.13949668 &amp; 1.56694829 &amp; -0.66717355 \\\ -0.97825977 &amp; 0.09190721 &amp; 1.59246789 &amp; -0.70611533 \\\ -1.10306273 &amp; 0.02854757 &amp; 1.58729045 &amp; -0.51277529 \end{pmatrix} $$</span></p>
  <p>How many hidden layers are in the model? It's not clear to me if each row (function) in the weight matrix is a &quot;layer&quot; or if the whole process of multiplying <span class="math-container">$LN(x_i)$</span> by <span class="math-container">$W_1$</span> is a single layer?</p>
  <p>As I understand it, <span class="math-container">$W_2$</span> is the output layer?</p>
  
Tags: <feedforward-neural-networks><multilayer-perceptrons><hidden-layers>
Title: How are hidden layers counted / semantically defined?
UserDisplayName: Grant Curell
ThreadText: # How are hidden layers counted / semantically defined?
  
  (Asked by: username_0 on 2023-11-02)
  
  I'm working my way through how LLMs work and I understand how things work but it's not clear to me exactly what is semantically defined as a "layer".
  
  Using the following FFN as an example:
  
  1. The weight matrices will be $W\_1$ of size $[4 \times 6]$ and $W\_2$ of size $[6 \times 4]$.
  2. The bias vectors will be $b\_1$ of size $[6]$ and $b\_2$ of size $[4]$.
  
  $$W\_1 = \begin{pmatrix} 0.1 & 0.2 & 0.3 & 0.4 & 0.5 & 0.6 \\\ 0.7 & 0.8 & 0.9 & 1.0 & 1.1 & 1.2 \\\ 1.3 & 1.4 & 1.5 & 1.6 & 1.7 & 1.8 \\\ 1.9 & 2.0 & 2.1 & 2.2 & 2.3 & 2.4 \end{pmatrix}$$
  
  $$b\_1 = \begin{pmatrix} 0.01 \\\ 0.02 \\\ 0.03 \\\ 0.04 \\\ 0.05 \\\ 0.06 \end{pmatrix}$$
  
  $$W\_2 = \begin{pmatrix} 0.1 & 0.7 & 1.3 & 1.9 \\\ 0.2 & 0.8 & 1.4 & 2.0 \\\ 0.3 & 0.9 & 1.5 & 2.1 \\\ 0.4 & 1.0 & 1.6 & 2.2 \\\ 0.5 & 1.1 & 1.7 & 2.3 \\\ 0.6 & 1.2 & 1.8 & 2.4 \end{pmatrix}$$
  
  $$b\_2 = \begin{pmatrix} 0.01 \\\ 0.02 \\\ 0.03 \\\ 0.04 \end{pmatrix}$$
  
  and just in case it matters, let's say that the input from layer normalization is:
  
  $$ \text{LN}(x\_i) = \begin{pmatrix} -1.03927142 & 0.13949668 & 1.56694829 & -0.66717355 \\\ -0.97825977 & 0.09190721 & 1.59246789 & -0.70611533 \\\ -1.10306273 & 0.02854757 & 1.58729045 & -0.51277529 \end{pmatrix} $$
  
  How many hidden layers are in the model? It's not clear to me if each row (function) in the weight matrix is a "layer" or if the whole process of multiplying $LN(x\_i)$ by $W\_1$ is a single layer?
  
  As I understand it, $W\_2$ is the output layer?
  
  **Tags:** feedforward-neural-networks, multilayer-perceptrons, hidden-layers
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-11-10. Score: 1 | Accepted answer
  
  In a FFN, a layer is a set of functions $f\_1, f\_2, \dots, f\_n$ like this $f\_j = \sigma(x\_1 \* w\_{j1} + \dots + x\_n \* w\_{jm})$ (sometimes called neurons), where $x\_i$ and $w\_i$ are respectively the inputs and parameters, and $\sigma$ is the activation function. So, $m$ is the number of inputs each function receives, which is equal to the number of parameters, while $n$ is the number of functions in the layer. You might have noticed that I wrote $w\_{j1}$ instead of $w\_1$ - this is intentional - each function has its own parameters.
  
  Now, can you write the computation of a layer, set of functions, as a matrix multiplication? Hint: yes
  
  Last, notice that $f\_j$ becomes $x\_j$ for the successive layer. So you can stack layers. Neural networks are just composite parametrized functions.
  
  
  ---
  
  ### Answer by username_2 on 2023-11-20. Score: 1
  
  For your example, $W\_2$, $b\_2$, and whatever activation function you use would collectively be considered the last layer. But for a more complicated architecture, such as a stack of LSTMs, the set of (matrix-vector multiplications + activations) that are necessary for a single LSTM gate might be considered a "layer".
  
  There's no formal definition of "layer". We call a group of calculations a "layer" whenever thinking about them as a single unit make it easier to reason about how the network as a whole works.
  
  
  ---
  
  