title: LLM Modules: Knowledge Transfer from a Large to a Small Model using
    Enhanced Cross-Attention
summary: In this work, we propose an architecture of LLM Modules that enables the
  transfer of knowledge from a large pre-trained model to a smaller model using
  an Enhanced Cross-Attention mechanism. In the proposed scheme, the Qwen2-1.5B
  model is frozen and its representations are passed through specially designed
  attention layers to the GPT-Neo-125M model, which is trained on limited
  computational resources. Experimental results on the Bespoke-Stratos-17k
  dataset demonstrate that after 15 epochs of training, the combined model
  generates responses comparable in quality to those obtained by distillation. We
  discuss the advantages of the modular approach, provide examples of input
  queries and comparative analysis, and outline prospects for further extension
  of the method.
paper_authors._id: 67ad7c47565fbcfa66777f5a
paper_authors.name: Konstantin Kolomeitsev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T08:21:29.819Z
paper_authors._id: 63df748df0c75dfb87621f93
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63df748df0c75dfb87621f93/Hzc3Du2BqnTP_8R9JsP-4.jpeg
paper_authors.fullname: Konstantin Kolomeitsev
paper_authors.type: user
paper_authors.user: kkolomeitsev
paper_title: LLM Modules: Knowledge Transfer from a Large to a Small Model using
    Enhanced Cross-Attention
paper_summary: In this work, we propose an architecture of LLM Modules that enables the
  transfer of knowledge from a large pre-trained model to a smaller model using
  an Enhanced Cross-Attention mechanism. In the proposed scheme, the Qwen2-1.5B
  model is frozen and its representations are passed through specially designed
  attention layers to the GPT-Neo-125M model, which is trained on limited
  computational resources. Experimental results on the Bespoke-Stratos-17k
  dataset demonstrate that after 15 epochs of training, the combined model
  generates responses comparable in quality to those obtained by distillation. We
  discuss the advantages of the modular approach, provide examples of input
  queries and comparative analysis, and outline prospects for further extension
  of the method.
paper_ai_summary: A large pre-trained model transfers knowledge to a smaller model via an enhanced cross-attention mechanism, achieving quality comparable to distillation after minimal training.
paper_ai_keywords: LLM Modules Enhanced Cross-Attention Qwen2-1.5B GPT-Neo-125M Bespoke-Stratos-17k dataset