title: Wonderful Matrices: Combining for a More Efficient and Effective
    Foundation Model Architecture
summary: In order to make the foundation model more efficient and effective, our idea
  is combining sequence transformation and state transformation. First, we prove
  the availability of rotary position embedding in the state space duality
  algorithm, which reduces the perplexity of the hybrid quadratic causal
  self-attention and state space duality by more than 4%, to ensure that the
  combining sequence transformation unifies position encoding. Second, we propose
  dynamic mask attention, which maintains 100% accuracy in the more challenging
  multi-query associative recall task, improving by more than 150% compared to
  quadratic causal self-attention and state space duality, to ensure that the
  combining sequence transformation selectively filters relevant information.
  Third, we design cross domain mixture of experts, which makes the computational
  speed of expert retrieval with more than 1024 experts 8 to 10 times faster than
  the mixture of experts, to ensure that the combining state transformation
  quickly retrieval mixture. Finally, we summarize these matrix algorithms that
  can form the foundation model: Wonderful Matrices, which can be a competitor to
  popular model architectures.
paper_authors._id: 6760f7212504d8a0fbf21ac9
paper_authors.name: Jingze Shi
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-17T03:59:29.876Z
paper_authors._id: 673ab3647afcea17eb4378fd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/673ab3647afcea17eb4378fd/YQB6zSH1LPxBMUYayIURi.png
paper_authors.fullname: Loser Cheems
paper_authors.type: user
paper_authors.user: JingzeShi
paper_authors._id: 6760f7212504d8a0fbf21aca
paper_authors.name: Bingheng Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:03:41.783Z
paper_authors._id: 66a0a9e7e81290f90f93b0a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66a0a9e7e81290f90f93b0a6/_HAsUv1eh0Im-aYZaa1C0.png
paper_authors.fullname: BinghengWu
paper_authors.type: user
paper_authors.user: wubingheng
paper_title: Wonderful Matrices: Combining for a More Efficient and Effective
    Foundation Model Architecture
paper_summary: In order to make the foundation model more efficient and effective, our idea
  is combining sequence transformation and state transformation. First, we prove
  the availability of rotary position embedding in the state space duality
  algorithm, which reduces the perplexity of the hybrid quadratic causal
  self-attention and state space duality by more than 4%, to ensure that the
  combining sequence transformation unifies position encoding. Second, we propose
  dynamic mask attention, which maintains 100% accuracy in the more challenging
  multi-query associative recall task, improving by more than 150% compared to
  quadratic causal self-attention and state space duality, to ensure that the
  combining sequence transformation selectively filters relevant information.
  Third, we design cross domain mixture of experts, which makes the computational
  speed of expert retrieval with more than 1024 experts 8 to 10 times faster than
  the mixture of experts, to ensure that the combining state transformation
  quickly retrieval mixture. Finally, we summarize these matrix algorithms that
  can form the foundation model: Wonderful Matrices, which can be a competitor to
  popular model architectures.
paper_ai_summary: A foundation model integrates sequence and state transformations using rotary position embedding, dynamic mask attention, and cross-domain mixture of experts, improving efficiency and accuracy.
paper_ai_keywords: rotary position embedding state space duality hybrid quadratic causal self-attention dynamic mask attention multi-query associative recall cross domain mixture of experts mixture of experts Wonderful Matrices