title: How Do Training Methods Influence the Utilization of Vision Models?
summary: Not all learnable parameters (e.g., weights) contribute equally to a neural
  network's decision function. In fact, entire layers' parameters can sometimes
  be reset to random values with little to no impact on the model's decisions. We
  revisit earlier studies that examined how architecture and task complexity
  influence this phenomenon and ask: is this phenomenon also affected by how we
  train the model? We conducted experimental evaluations on a diverse set of
  ImageNet-1k classification models to explore this, keeping the architecture and
  training data constant but varying the training pipeline. Our findings reveal
  that the training method strongly influences which layers become critical to
  the decision function for a given task. For example, improved training regimes
  and self-supervised training increase the importance of early layers while
  significantly under-utilizing deeper layers. In contrast, methods such as
  adversarial training display an opposite trend. Our preliminary results extend
  previous findings, offering a more nuanced understanding of the inner mechanics
  of neural networks.
    Code: https://github.com/paulgavrikov/layer_criticality
paper_authors._id: 6716249236c0c14b5b1dc6a5
paper_authors.name: Paul Gavrikov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:26:52.565Z
paper_authors._id: 6266c07e7a1f5a1562c4113b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6266c07e7a1f5a1562c4113b/GEt70X7-UmNIbECDA0eVs.jpeg
paper_authors.fullname: Paul Gavrikov
paper_authors.type: user
paper_authors.user: paulgavrikov
paper_authors._id: 6716249236c0c14b5b1dc6a6
paper_authors.name: Shashank Agnihotri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:26:58.209Z
paper_authors._id: 67162cabea37ca179f5965c9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7lKl5w8ecex4eP2I89MPz.png
paper_authors.fullname: Shashank Agnihotri
paper_authors.type: user
paper_authors.user: shashankskagnihotri
paper_authors._id: 6716249236c0c14b5b1dc6a7
paper_authors.name: Margret Keuper
paper_authors._id: 6716249236c0c14b5b1dc6a8
paper_authors.name: Janis Keuper
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-14T09:00:05.532Z
paper_authors._id: 6519013e3fe8cc63b4c9b8d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6519013e3fe8cc63b4c9b8d3/ShO-DM3nWhKTeg_qJ3p0P.jpeg
paper_authors.fullname: Janis Keuper
paper_authors.type: user
paper_authors.user: jkHSO
paper_title: How Do Training Methods Influence the Utilization of Vision Models?
paper_summary: Not all learnable parameters (e.g., weights) contribute equally to a neural
  network's decision function. In fact, entire layers' parameters can sometimes
  be reset to random values with little to no impact on the model's decisions. We
  revisit earlier studies that examined how architecture and task complexity
  influence this phenomenon and ask: is this phenomenon also affected by how we
  train the model? We conducted experimental evaluations on a diverse set of
  ImageNet-1k classification models to explore this, keeping the architecture and
  training data constant but varying the training pipeline. Our findings reveal
  that the training method strongly influences which layers become critical to
  the decision function for a given task. For example, improved training regimes
  and self-supervised training increase the importance of early layers while
  significantly under-utilizing deeper layers. In contrast, methods such as
  adversarial training display an opposite trend. Our preliminary results extend
  previous findings, offering a more nuanced understanding of the inner mechanics
  of neural networks.
    Code: https://github.com/paulgavrikov/layer_criticality
paper_ai_keywords: RAG (Retrieval-Augmented Generation) weights ImageNet-1k classification models decision function architecture task complexity training method region criticality training pipeline self-supervised training adversarial training early layers deeper layers generated sequence inner mechanics