title: Composable Function-preserving Expansions for Transformer Architectures
summary: Training state-of-the-art neural networks requires a high cost in terms of
  compute and time. Model scale is recognized to be a critical factor to achieve
  and improve the state-of-the-art. Increasing the scale of a neural network
  normally requires restarting from scratch by randomly initializing all the
  parameters of the model, as this implies a change of architecture's parameters
  that does not allow for a straightforward transfer of knowledge from smaller
  size models. In this work, we propose six composable transformations to
  incrementally increase the size of transformer-based neural networks while
  preserving functionality, allowing to expand the capacity of the model as
  needed. We provide proof of exact function preservation under minimal
  initialization constraints for each transformation. The proposed methods may
  enable efficient training pipelines for larger and more powerful models by
  progressively expanding the architecture throughout training.
paper_authors._id: 64d9b8145f4814f7c35643eb
paper_authors.name: Andrea Gesmundo
paper_authors._id: 64d9b8145f4814f7c35643ec
paper_authors.name: Kaitlin Maile
paper_title: Composable Function-preserving Expansions for Transformer Architectures
paper_summary: Training state-of-the-art neural networks requires a high cost in terms of
  compute and time. Model scale is recognized to be a critical factor to achieve
  and improve the state-of-the-art. Increasing the scale of a neural network
  normally requires restarting from scratch by randomly initializing all the
  parameters of the model, as this implies a change of architecture's parameters
  that does not allow for a straightforward transfer of knowledge from smaller
  size models. In this work, we propose six composable transformations to
  incrementally increase the size of transformer-based neural networks while
  preserving functionality, allowing to expand the capacity of the model as
  needed. We provide proof of exact function preservation under minimal
  initialization constraints for each transformation. The proposed methods may
  enable efficient training pipelines for larger and more powerful models by
  progressively expanding the architecture throughout training.
paper_ai_summary: The research introduces methods to incrementally enlarge transformer-based neural networks while preserving their functionality, enabling efficient training of larger models.
paper_ai_keywords: neural networks model scale parameter initialization transformers architecture expansion function preservation training pipelines