title: Scaling TransNormer to 175 Billion Parameters
summary: We present TransNormerLLM, the first linear attention-based Large Language
  Model (LLM) that outperforms conventional softmax attention-based models in
  terms of both accuracy and efficiency. TransNormerLLM evolves from the previous
  linear attention architecture TransNormer by making advanced modifications that
  include positional embedding, linear attention acceleration, gating mechanism,
  tensor normalization, inference acceleration and stabilization. Specifically,
  we use LRPE together with an exponential decay to avoid attention dilution
  issues while allowing the model to retain global interactions between tokens.
  Additionally, we propose Lightning Attention, a cutting-edge technique that
  accelerates linear attention by more than twice in runtime and reduces memory
  usage by a remarkable four times. To further enhance the performance of
  TransNormer, we leverage a gating mechanism to smooth training and a new tensor
  normalization scheme to accelerate the model, resulting in an impressive
  acceleration of over 20%. Furthermore, we have developed a robust inference
  algorithm that ensures numerical stability and consistent inference speed,
  regardless of the sequence length, showcasing superior efficiency during both
  training and inference stages. Scalability is at the heart of our model's
  design, enabling seamless deployment on large-scale clusters and facilitating
  expansion to even more extensive models, all while maintaining outstanding
  performance metrics. Rigorous validation of our model design is achieved
  through a series of comprehensive experiments on our self-collected corpus,
  boasting a size exceeding 6TB and containing over 2 trillion tokens. To ensure
  data quality and relevance, we implement a new self-cleaning strategy to filter
  our collected data. Our pre-trained models will be released to foster community
  advancements in efficient LLMs.
paper_authors._id: 64c317511920bd982085c652
paper_authors.name: Zhen Qin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:48:04.814Z
paper_authors._id: 642e63a53c2cf43f6d6dc5ce
paper_authors.avatarUrl: /avatars/dfd78c8d55485c22be6e616670a633e5.svg
paper_authors.fullname: zhenqin
paper_authors.type: user
paper_authors.user: Doreamonzzz
paper_authors._id: 64c317511920bd982085c653
paper_authors.name: Dong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-13T08:32:00.398Z
paper_authors._id: 61cd713f78c1c6b489a231fc
paper_authors.avatarUrl: /avatars/af261b85bbd10a1182372ffc459640b8.svg
paper_authors.fullname: Dong Li
paper_authors.type: user
paper_authors.user: liddalidd
paper_authors._id: 64c317511920bd982085c654
paper_authors.name: Weigao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-28T08:18:48.288Z
paper_authors._id: 6246bb33da617c00b48e4d92
paper_authors.avatarUrl: /avatars/0304a9f6eb7f5dee4d933d03222f94e9.svg
paper_authors.fullname: Weigao Sun
paper_authors.type: user
paper_authors.user: weigao266
paper_authors._id: 64c317511920bd982085c655
paper_authors.name: Weixuan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-28T08:19:08.964Z
paper_authors._id: 640c4d2d536d9fe0f000b209
paper_authors.avatarUrl: /avatars/419e59b78f7fc7f6b3ac343398d11ec1.svg
paper_authors.fullname: Weixuan Sun
paper_authors.type: user
paper_authors.user: weixuansun
paper_authors._id: 64c317511920bd982085c656
paper_authors.name: Xuyang Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:35:15.641Z
paper_authors._id: 642e4d4d6748dd4f8eeb7732
paper_authors.avatarUrl: /avatars/fd911e9143d1a7aedd21a7d611543fcc.svg
paper_authors.fullname: Xuyang Shen
paper_authors.type: user
paper_authors.user: Ryan1122
paper_authors._id: 64c317511920bd982085c657
paper_authors.name: Xiaodong Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-28T08:20:11.394Z
paper_authors._id: 64638bddc615cbc12447f9f1
paper_authors.avatarUrl: /avatars/52910d1717451ff983f745322e5850dd.svg
paper_authors.fullname: Xiaodong Han
paper_authors.type: user
paper_authors.user: Hannnnnxd
paper_authors._id: 64c317511920bd982085c658
paper_authors.name: Yunshen Wei
paper_authors._id: 64c317511920bd982085c659
paper_authors.name: Baohong Lv
paper_authors._id: 64c317511920bd982085c65a
paper_authors.name: Fei Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-25T09:45:13.208Z
paper_authors._id: 65fed45b08d35929362dd651
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65fed45b08d35929362dd651/KLMxsyRN6_HhCZP1iDw6K.png
paper_authors.fullname: FeiYuan
paper_authors.type: user
paper_authors.user: FeYuan
paper_authors._id: 64c317511920bd982085c65b
paper_authors.name: Xiao Luo
paper_authors._id: 64c317511920bd982085c65c
paper_authors.name: Yu Qiao
paper_authors._id: 64c317511920bd982085c65d
paper_authors.name: Yiran Zhong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-20T09:32:38.615Z
paper_authors._id: 64c525e4d68946edad6c7067
paper_authors.avatarUrl: /avatars/1b108661634af602717a4ab4b66a151f.svg
paper_authors.fullname: Yiran Zhong
paper_authors.type: user
paper_authors.user: IanZhong
paper_title: Scaling TransNormer to 175 Billion Parameters
paper_summary: We present TransNormerLLM, the first linear attention-based Large Language
  Model (LLM) that outperforms conventional softmax attention-based models in
  terms of both accuracy and efficiency. TransNormerLLM evolves from the previous
  linear attention architecture TransNormer by making advanced modifications that
  include positional embedding, linear attention acceleration, gating mechanism,
  tensor normalization, inference acceleration and stabilization. Specifically,
  we use LRPE together with an exponential decay to avoid attention dilution
  issues while allowing the model to retain global interactions between tokens.
  Additionally, we propose Lightning Attention, a cutting-edge technique that
  accelerates linear attention by more than twice in runtime and reduces memory
  usage by a remarkable four times. To further enhance the performance of
  TransNormer, we leverage a gating mechanism to smooth training and a new tensor
  normalization scheme to accelerate the model, resulting in an impressive
  acceleration of over 20%. Furthermore, we have developed a robust inference
  algorithm that ensures numerical stability and consistent inference speed,
  regardless of the sequence length, showcasing superior efficiency during both
  training and inference stages. Scalability is at the heart of our model's
  design, enabling seamless deployment on large-scale clusters and facilitating
  expansion to even more extensive models, all while maintaining outstanding
  performance metrics. Rigorous validation of our model design is achieved
  through a series of comprehensive experiments on our self-collected corpus,
  boasting a size exceeding 6TB and containing over 2 trillion tokens. To ensure
  data quality and relevance, we implement a new self-cleaning strategy to filter
  our collected data. Our pre-trained models will be released to foster community
  advancements in efficient LLMs.
paper_ai_keywords: linear attention Large Language Model (LLM) softmax attention positional embedding linear attention acceleration gating mechanism tensor normalization inference acceleration attention dilution exponential decay global interactions Lightning Attention numerical stability sequence length scalability pre-trained models self-collected corpus self-cleaning strategy efficient LLMs