title: SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and
    Composition of Experts
summary: Monolithic large language models (LLMs) like GPT-4 have paved the way for
  modern generative AI applications. Training, serving, and maintaining
  monolithic LLMs at scale, however, remains prohibitively expensive and
  challenging. The disproportionate increase in compute-to-memory ratio of modern
  AI accelerators have created a memory wall, necessitating new methods to deploy
  AI. Composition of Experts (CoE) is an alternative modular approach that lowers
  the cost and complexity of training and serving. However, this approach
  presents two key challenges when using conventional hardware: (1) without fused
  operations, smaller models have lower operational intensity, which makes high
  utilization more challenging to achieve; and (2) hosting a large number of
  models can be either prohibitively expensive or slow when dynamically switching
  between them.
    In this paper, we describe how combining CoE, streaming dataflow, and a
  three-tier memory system scales the AI memory wall. We describe Samba-CoE, a
  CoE system with 150 experts and a trillion total parameters. We deploy
  Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) - a
  commercial dataflow accelerator architecture that has been co-designed for
  enterprise inference and training applications. The chip introduces a new
  three-tier memory system with on-chip distributed SRAM, on-package HBM, and
  off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out
  over multiple sockets. We demonstrate speedups ranging from 2x to 13x on
  various benchmarks running on eight RDU sockets compared with an unfused
  baseline. We show that for CoE inference deployments, the 8-socket RDU Node
  reduces machine footprint by up to 19x, speeds up model switching time by 15x
  to 31x, and achieves an overall speedup of 3.7x over a DGX H100 and 6.6x over a
  DGX A100.
paper_authors._id: 6642e376ccd85d80ae1f5f63
paper_authors.name: Raghu Prabhakar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T08:14:17.850Z
paper_authors._id: 65e280b168e17130cb4b9c65
paper_authors.avatarUrl: /avatars/2a942ce7a45b5fe74c01eb86833812f0.svg
paper_authors.fullname: Raghu Prabhakar
paper_authors.type: user
paper_authors.user: raghup17
paper_authors._id: 6642e376ccd85d80ae1f5f64
paper_authors.name: Ram Sivaramakrishnan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-24T09:31:43.742Z
paper_authors._id: 665035f801646a66be790a49
paper_authors.avatarUrl: /avatars/fa708c3c18d1ef58ace6c9e56b7216e3.svg
paper_authors.fullname: Ram Sivaramakrishnan
paper_authors.type: user
paper_authors.user: ramas1323
paper_authors._id: 6642e376ccd85d80ae1f5f65
paper_authors.name: Darshan Gandhi
paper_authors._id: 6642e376ccd85d80ae1f5f66
paper_authors.name: Yun Du
paper_authors._id: 6642e376ccd85d80ae1f5f67
paper_authors.name: Mingran Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T08:15:51.931Z
paper_authors._id: 6438fea00a97f2ff4e5d0c98
paper_authors.avatarUrl: /avatars/5328f519ff4a831efa5ba97f8391a31e.svg
paper_authors.fullname: Mingran Wang
paper_authors.type: user
paper_authors.user: mingranw
paper_authors._id: 6642e376ccd85d80ae1f5f68
paper_authors.name: Xiangyu Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T08:16:03.219Z
paper_authors._id: 644d07456d84e25fad74172e
paper_authors.avatarUrl: /avatars/acea0f0d54a6285ddfcb0ad1b7a494d7.svg
paper_authors.fullname: songxiangyu
paper_authors.type: user
paper_authors.user: sabarela
paper_authors._id: 6642e376ccd85d80ae1f5f69
paper_authors.name: Kejie Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T08:16:16.242Z
paper_authors._id: 614166546746c507644e53a3
paper_authors.avatarUrl: /avatars/6bbb57d46ba520a8e2726f9ac6d2d8ed.svg
paper_authors.fullname: Kejie Zhang
paper_authors.type: user
paper_authors.user: kejiezh
paper_authors._id: 6642e376ccd85d80ae1f5f6a
paper_authors.name: Tianren Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T08:16:23.228Z
paper_authors._id: 65a9d6180150f64adf7be16c
paper_authors.avatarUrl: /avatars/1f8e1d66dcc5bdf619a9fdc28c11211c.svg
paper_authors.fullname: Tianren Gao
paper_authors.type: user
paper_authors.user: TerryGao87
paper_authors._id: 6642e376ccd85d80ae1f5f6b
paper_authors.name: Angela Wang
paper_authors._id: 6642e376ccd85d80ae1f5f6c
paper_authors.name: Karen Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T08:16:49.193Z
paper_authors._id: 66289bc249d2e51ea8099d5b
paper_authors.avatarUrl: /avatars/3b680e06dc3d01c534a782c167e54bc5.svg
paper_authors.fullname: Li
paper_authors.type: user
paper_authors.user: KarenLi
paper_authors._id: 6642e376ccd85d80ae1f5f6d
paper_authors.name: Yongning Sheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-15T10:43:05.825Z
paper_authors._id: 6643b625b5a21591716342c1
paper_authors.avatarUrl: /avatars/0def9fb5d67b1105a010c8320c809130.svg
paper_authors.fullname: Yongning Sheng
paper_authors.type: user
paper_authors.user: yongningsheng
paper_authors._id: 6642e376ccd85d80ae1f5f6e
paper_authors.name: Joshua Brot
paper_authors._id: 6642e376ccd85d80ae1f5f6f
paper_authors.name: Denis Sokolov
paper_authors._id: 6642e376ccd85d80ae1f5f70
paper_authors.name: Apurv Vivek
paper_authors._id: 6642e376ccd85d80ae1f5f71
paper_authors.name: Calvin Leung
paper_authors._id: 6642e376ccd85d80ae1f5f72
paper_authors.name: Arjun Sabnis
paper_authors._id: 6642e376ccd85d80ae1f5f73
paper_authors.name: Jiayu Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T08:17:44.200Z
paper_authors._id: 642fabe6c953ca48ccd40280
paper_authors.avatarUrl: /avatars/7c4d98b47da1e7aaa054bacb0ca572d9.svg
paper_authors.fullname: Jiayu Bai
paper_authors.type: user
paper_authors.user: checkRaiseOnCloud
paper_authors._id: 6642e376ccd85d80ae1f5f74
paper_authors.name: Tuowen Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T08:17:50.450Z
paper_authors._id: 663ab4bfade1a39663c78dff
paper_authors.avatarUrl: /avatars/d9769941408ed6fb2373091e34b056c2.svg
paper_authors.fullname: Tuowen Zhao
paper_authors.type: user
paper_authors.user: tuowenz
paper_authors._id: 6642e376ccd85d80ae1f5f75
paper_authors.name: Mark Gottscho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T08:17:58.960Z
paper_authors._id: 656d47a509d123964bf3eb49
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/fM5XouKzE9P6J48TbLV5A.jpeg
paper_authors.fullname: Mark Gottscho
paper_authors.type: user
paper_authors.user: mgottscho
paper_authors._id: 6642e376ccd85d80ae1f5f76
paper_authors.name: David Jackson
paper_authors._id: 6642e376ccd85d80ae1f5f77
paper_authors.name: Mark Luttrell
paper_authors._id: 6642e376ccd85d80ae1f5f78
paper_authors.name: Manish K. Shah
paper_authors._id: 6642e376ccd85d80ae1f5f79
paper_authors.name: Edison Chen
paper_authors._id: 6642e376ccd85d80ae1f5f7a
paper_authors.name: Kaizhao Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T08:18:26.156Z
paper_authors._id: 62140dcdcf7928035e8135ad
paper_authors.avatarUrl: /avatars/146a8ac3fb476e79b6503373df0f38c1.svg
paper_authors.fullname: Kaizhao Liang
paper_authors.type: user
paper_authors.user: kz919
paper_authors._id: 6642e376ccd85d80ae1f5f7b
paper_authors.name: Swayambhoo Jain
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-24T09:31:47.085Z
paper_authors._id: 645d7ed4f4f49de580a00fd1
paper_authors.avatarUrl: /avatars/a47839f93a01a64411987f075d167370.svg
paper_authors.fullname: Swayambhoo Jain
paper_authors.type: user
paper_authors.user: swayambhooj
paper_authors._id: 6642e376ccd85d80ae1f5f7c
paper_authors.name: Urmish Thakker
paper_authors._id: 6642e376ccd85d80ae1f5f7d
paper_authors.name: Dawei Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T08:19:27.846Z
paper_authors._id: 65f75e3df89ec850f7ac7f01
paper_authors.avatarUrl: /avatars/4ec9b4a4e42207907acbb87af17e7b1e.svg
paper_authors.fullname: Dawei Huang
paper_authors.type: user
paper_authors.user: daweih
paper_authors._id: 6642e376ccd85d80ae1f5f7e
paper_authors.name: Sumti Jairath
paper_authors._id: 6642e376ccd85d80ae1f5f7f
paper_authors.name: Kevin J. Brown
paper_authors._id: 6642e376ccd85d80ae1f5f80
paper_authors.name: Kunle Olukotun
paper_title: SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and
    Composition of Experts
paper_summary: Monolithic large language models (LLMs) like GPT-4 have paved the way for
  modern generative AI applications. Training, serving, and maintaining
  monolithic LLMs at scale, however, remains prohibitively expensive and
  challenging. The disproportionate increase in compute-to-memory ratio of modern
  AI accelerators have created a memory wall, necessitating new methods to deploy
  AI. Composition of Experts (CoE) is an alternative modular approach that lowers
  the cost and complexity of training and serving. However, this approach
  presents two key challenges when using conventional hardware: (1) without fused
  operations, smaller models have lower operational intensity, which makes high
  utilization more challenging to achieve; and (2) hosting a large number of
  models can be either prohibitively expensive or slow when dynamically switching
  between them.
    In this paper, we describe how combining CoE, streaming dataflow, and a
  three-tier memory system scales the AI memory wall. We describe Samba-CoE, a
  CoE system with 150 experts and a trillion total parameters. We deploy
  Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) - a
  commercial dataflow accelerator architecture that has been co-designed for
  enterprise inference and training applications. The chip introduces a new
  three-tier memory system with on-chip distributed SRAM, on-package HBM, and
  off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out
  over multiple sockets. We demonstrate speedups ranging from 2x to 13x on
  various benchmarks running on eight RDU sockets compared with an unfused
  baseline. We show that for CoE inference deployments, the 8-socket RDU Node
  reduces machine footprint by up to 19x, speeds up model switching time by 15x
  to 31x, and achieves an overall speedup of 3.7x over a DGX H100 and 6.6x over a
  DGX A100.
paper_ai_keywords: Text Research Large Language Models (LLMs) Generative AI applications Monolithic LLMs Training Serving Maintenance Compute-to-memory ratio AI accelerators Composition of Experts (CoE) Modular approach Operational intensity Utilization Dynamic switching Streaming Dataflow Three-tier memory system Samba-CoE Experts Parameters SambaNova SN40L Reconfigurable Dataflow Unit (RDU) On-chip distributed SRAM On-package HBM Off-package DDR DRAM Inter-RDU network Speedup Benchmarks Baseline Machine footprint Model switching time DGX H100 DGX A100