Answers.Body: <p>For your concerned transformer-based models such as the decoder-only GPT model, hierarchical feature extraction is achieved through the <em>iterative</em> application of masked multi-head self-attention (MHA) blocks, residual connection and layer normalization (ADD, NORM), and feed-forward networks (FFN).</p>
  <p>MHAs allow the model to focus on different aspects of the input sequence simultaneously, where each head can capture different patterns and relative token importance via <em>linear</em> projections with learned weight matrices to generate their corresponding Query/Key/Value matrices and <em>scaled</em> dot-product attention scores. The attention heads in the initial layers might capture simple syntactic relationships such as adjacent word types, PoS tags, basic phrases and basic syntactic rules, while later layers' MHAs are responsible for understanding high-level abstractions, long range indexical dependencies, narratives and overall context such as the coherence of a whole paragraph via same mechanism but with integrated and enriched contextual embeddings output from early layers.</p>
  <p>The FFNs further transform local patterns identified by their previous MHAs into more abstract <em>nonlinear</em> relational features usually through a two-step linear transformation with a non-linear activation in between. Also they're followed by residual connection (adding the original input to the output) which helps in stabilizing training and preserving useful information from previous layers and they can be tailored to specific requirements of any downstream task by adjusting its architecture. This sequential process allows the model to progressively build more complex and abstract representations enabling it to understand and generate coherent and refined high quality text. Of course all these are just very high level overview for the quite complicated GPT transformer model but the hierarchical iterative learning step is clear.</p>
  <p>In summary for deep learning models or architectures such as CNN, RNN, Transformers, Transfer Learning, etc, learning multiple levels of composition is indeed equivalent to hierarchical processing or hierarchical feature extraction since most neural networks are essentially of the nested recursive function composition form <span class="math-container">$f(\mathbf{x};\mathbf{\theta})=f_{L}(f_{L-1}(...f_1(\mathbf{x})))$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-29T01:13:37.277Z
Answers.LastActivityDate: 2024-07-29T02:47:45.233Z
Answers.LastEditDate: 2024-07-29T02:47:45.233Z
Answers.UserDisplayName: cinch
Body: <p>CNNs are a famous example of hierarchical processing. Lower-level features such as edges are detected in the earlier layers while higher-level features such as the presence of a face are detected deeper in the network.</p>
  <p>All deep learning architectures share a common thing (excerpt taken from the <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer">Deep Learning book</a>):</p>
  <blockquote>
  <p>The modern term “deep learning” goes beyond the neuroscientiﬁc
  perspective on the current breed of machine learning models. It
  appeals to a more general principle of <strong>learning multiple levels of
  composition</strong>, which can be applied in machine learning frameworks that
  are not necessarily neurally inspired.</p>
  </blockquote>
  <p>My question is the following:</p>
  <p>Is learning multiple levels of composition equivalent to hierarchical processing/hierarchical feature extraction?</p>
  <p>If yes, what kind of hierarchies are extracted in fields other than computer vision? For example, what kind of features are extracted hierarchically from a transformer-based architecture or a graph neural network?</p>
  
Tags: <deep-learning><feature-extraction><representation-learning>
Title: Do all deep learning architectures learn features hierarchically?
UserDisplayName: Antonios Sarikas
ThreadText: # Do all deep learning architectures learn features hierarchically?
  
  (Asked by: username_0 on 2024-07-28)
  
  CNNs are a famous example of hierarchical processing. Lower-level features such as edges are detected in the earlier layers while higher-level features such as the presence of a face are detected deeper in the network.
  
  All deep learning architectures share a common thing (excerpt taken from the [Deep Learning book](https://www.deeplearningbook.org/)):
  
  > The modern term “deep learning” goes beyond the neuroscientiﬁc
  > perspective on the current breed of machine learning models. It
  > appeals to a more general principle of **learning multiple levels of
  > composition**, which can be applied in machine learning frameworks that
  > are not necessarily neurally inspired.
  
  My question is the following:
  
  Is learning multiple levels of composition equivalent to hierarchical processing/hierarchical feature extraction?
  
  If yes, what kind of hierarchies are extracted in fields other than computer vision? For example, what kind of features are extracted hierarchically from a transformer-based architecture or a graph neural network?
  
  **Tags:** deep-learning, feature-extraction, representation-learning
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-29. Score: 0
  
  For your concerned transformer-based models such as the decoder-only GPT model, hierarchical feature extraction is achieved through the *iterative* application of masked multi-head self-attention (MHA) blocks, residual connection and layer normalization (ADD, NORM), and feed-forward networks (FFN).
  
  MHAs allow the model to focus on different aspects of the input sequence simultaneously, where each head can capture different patterns and relative token importance via *linear* projections with learned weight matrices to generate their corresponding Query/Key/Value matrices and *scaled* dot-product attention scores. The attention heads in the initial layers might capture simple syntactic relationships such as adjacent word types, PoS tags, basic phrases and basic syntactic rules, while later layers' MHAs are responsible for understanding high-level abstractions, long range indexical dependencies, narratives and overall context such as the coherence of a whole paragraph via same mechanism but with integrated and enriched contextual embeddings output from early layers.
  
  The FFNs further transform local patterns identified by their previous MHAs into more abstract *nonlinear* relational features usually through a two-step linear transformation with a non-linear activation in between. Also they're followed by residual connection (adding the original input to the output) which helps in stabilizing training and preserving useful information from previous layers and they can be tailored to specific requirements of any downstream task by adjusting its architecture. This sequential process allows the model to progressively build more complex and abstract representations enabling it to understand and generate coherent and refined high quality text. Of course all these are just very high level overview for the quite complicated GPT transformer model but the hierarchical iterative learning step is clear.
  
  In summary for deep learning models or architectures such as CNN, RNN, Transformers, Transfer Learning, etc, learning multiple levels of composition is indeed equivalent to hierarchical processing or hierarchical feature extraction since most neural networks are essentially of the nested recursive function composition form $f(\mathbf{x};\mathbf{\theta})=f\_{L}(f\_{L-1}(...f\_1(\mathbf{x})))$.
  
  
  ---
  
  