Growing Transformers: Modular Composition and Layer-wise Expansion on a
  Frozen Substrate
The prevailing paradigm for scaling large language models (LLMs) involves
monolithic, end-to-end training, a resource-intensive process that lacks
flexibility. This paper explores an alternative, constructive approach to model
development, built upon the foundation of non-trainable, deterministic input
embeddings. In prior [1], we established that high-level semantic reasoning can
emerge in Transformers using frozen embeddings derived from the visual
structure of Unicode glyphs. Here, we demonstrate that this fixed
representational substrate acts as a universal "docking port," enabling two
powerful and efficient scaling paradigms: seamless modular composition and
progressive layer-wise growth.
  First, we show that specialist models trained on disparate datasets (e.g.,
Russian and Chinese text) can be merged into a single, more capable
Mixture-of-Experts (MoE) model, post-training, with zero architectural
modification. This is achieved by simply averaging their output logits. The
resulting MoE model exhibits immediate performance improvements on reasoning
benchmarks like MMLU, surpassing its constituent experts without catastrophic
forgetting. Second, we introduce a layer-wise constructive training
methodology, where a deep Transformer is "grown" by progressively stacking and
training one layer at a time. This method demonstrates stable convergence and a
clear correlation between model depth and the emergence of complex reasoning
abilities, such as those required for SQuAD.
  Our findings suggest a paradigm shift from monolithic optimization towards a
more biological or constructive model of AI development, where complexity is
built incrementally and modules can be composed freely. This opens new avenues
for resource-efficient scaling, continual learning, and a more democratized
ecosystem for building powerful AI systems. We release all code and models to
facilitate further research.
6870d30fc8391850d60978fe
A. Bochkov
claimed_verified
2025-07-11T13:12:34.561Z
661d507b4fe8c7a85ab96e3e
/avatars/6e7564331d0f1696e94377784a4b0e9b.svg
Andrey
user
Bochkov
Growing Transformers: Modular Composition and Layer-wise Expansion on a
  Frozen Substrate
The prevailing paradigm for scaling large language models (LLMs) involves
monolithic, end-to-end training, a resource-intensive process that lacks
flexibility. This paper explores an alternative, constructive approach to model
development, built upon the foundation of non-trainable, deterministic input
embeddings. In prior [1], we established that high-level semantic reasoning can
emerge in Transformers using frozen embeddings derived from the visual
structure of Unicode glyphs. Here, we demonstrate that this fixed
representational substrate acts as a universal "docking port," enabling two
powerful and efficient scaling paradigms: seamless modular composition and
progressive layer-wise growth.
  First, we show that specialist models trained on disparate datasets (e.g.,
Russian and Chinese text) can be merged into a single, more capable
Mixture-of-Experts (MoE) model, post-training, with zero architectural
modification. This is achieved by simply averaging their output logits. The
resulting MoE model exhibits immediate performance improvements on reasoning
benchmarks like MMLU, surpassing its constituent experts without catastrophic
forgetting. Second, we introduce a layer-wise constructive training
methodology, where a deep Transformer is "grown" by progressively stacking and
training one layer at a time. This method demonstrates stable convergence and a
clear correlation between model depth and the emergence of complex reasoning
abilities, such as those required for SQuAD.
  Our findings suggest a paradigm shift from monolithic optimization towards a
more biological or constructive model of AI development, where complexity is
built incrementally and modules can be composed freely. This opens new avenues
for resource-efficient scaling, continual learning, and a more democratized
ecosystem for building powerful AI systems. We release all code and models to
facilitate further research.
Transformers with frozen embeddings enable efficient scaling through modular composition and layer-wise growth, improving performance on reasoning tasks without catastrophic forgetting.
Transformers frozen embeddings Mixture-of-Experts (MoE) modular composition layer-wise growth SQuAD MMLU catastrophic forgetting