SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and
  Composition of Experts
Monolithic large language models (LLMs) like GPT-4 have paved the way for
modern generative AI applications. Training, serving, and maintaining
monolithic LLMs at scale, however, remains prohibitively expensive and
challenging. The disproportionate increase in compute-to-memory ratio of modern
AI accelerators have created a memory wall, necessitating new methods to deploy
AI. Composition of Experts (CoE) is an alternative modular approach that lowers
the cost and complexity of training and serving. However, this approach
presents two key challenges when using conventional hardware: (1) without fused
operations, smaller models have lower operational intensity, which makes high
utilization more challenging to achieve; and (2) hosting a large number of
models can be either prohibitively expensive or slow when dynamically switching
between them.
  In this paper, we describe how combining CoE, streaming dataflow, and a
three-tier memory system scales the AI memory wall. We describe Samba-CoE, a
CoE system with 150 experts and a trillion total parameters. We deploy
Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) - a
commercial dataflow accelerator architecture that has been co-designed for
enterprise inference and training applications. The chip introduces a new
three-tier memory system with on-chip distributed SRAM, on-package HBM, and
off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out
over multiple sockets. We demonstrate speedups ranging from 2x to 13x on
various benchmarks running on eight RDU sockets compared with an unfused
baseline. We show that for CoE inference deployments, the 8-socket RDU Node
reduces machine footprint by up to 19x, speeds up model switching time by 15x
to 31x, and achieves an overall speedup of 3.7x over a DGX H100 and 6.6x over a
DGX A100.
6642e376ccd85d80ae1f5f63
Raghu Prabhakar
admin_assigned
2024-05-14T08:14:17.850Z
65e280b168e17130cb4b9c65
/avatars/2a942ce7a45b5fe74c01eb86833812f0.svg
Raghu Prabhakar
user
raghup17
6642e376ccd85d80ae1f5f64
Ram Sivaramakrishnan
claimed_verified
2024-05-24T09:31:43.742Z
665035f801646a66be790a49
/avatars/fa708c3c18d1ef58ace6c9e56b7216e3.svg
Ram Sivaramakrishnan
user
ramas1323
6642e376ccd85d80ae1f5f65
Darshan Gandhi
6642e376ccd85d80ae1f5f66
Yun Du
6642e376ccd85d80ae1f5f67
Mingran Wang
admin_assigned
2024-05-14T08:15:51.931Z
6438fea00a97f2ff4e5d0c98
/avatars/5328f519ff4a831efa5ba97f8391a31e.svg
Mingran Wang
user
mingranw
6642e376ccd85d80ae1f5f68
Xiangyu Song
admin_assigned
2024-05-14T08:16:03.219Z
644d07456d84e25fad74172e
/avatars/acea0f0d54a6285ddfcb0ad1b7a494d7.svg
songxiangyu
user
sabarela
6642e376ccd85d80ae1f5f69
Kejie Zhang
admin_assigned
2024-05-14T08:16:16.242Z
614166546746c507644e53a3
/avatars/6bbb57d46ba520a8e2726f9ac6d2d8ed.svg
Kejie Zhang
user
kejiezh
6642e376ccd85d80ae1f5f6a
Tianren Gao
admin_assigned
2024-05-14T08:16:23.228Z
65a9d6180150f64adf7be16c
/avatars/1f8e1d66dcc5bdf619a9fdc28c11211c.svg
Tianren Gao
user
TerryGao87
6642e376ccd85d80ae1f5f6b
Angela Wang
6642e376ccd85d80ae1f5f6c
Karen Li
admin_assigned
2024-05-14T08:16:49.193Z
66289bc249d2e51ea8099d5b
/avatars/3b680e06dc3d01c534a782c167e54bc5.svg
Li
user
KarenLi
6642e376ccd85d80ae1f5f6d
Yongning Sheng
claimed_verified
2024-05-15T10:43:05.825Z
6643b625b5a21591716342c1
/avatars/0def9fb5d67b1105a010c8320c809130.svg
Yongning Sheng
user
yongningsheng
6642e376ccd85d80ae1f5f6e
Joshua Brot
6642e376ccd85d80ae1f5f6f
Denis Sokolov
6642e376ccd85d80ae1f5f70
Apurv Vivek
6642e376ccd85d80ae1f5f71
Calvin Leung
6642e376ccd85d80ae1f5f72
Arjun Sabnis
6642e376ccd85d80ae1f5f73
Jiayu Bai
admin_assigned
2024-05-14T08:17:44.200Z
642fabe6c953ca48ccd40280
/avatars/7c4d98b47da1e7aaa054bacb0ca572d9.svg
Jiayu Bai
user
checkRaiseOnCloud
6642e376ccd85d80ae1f5f74
Tuowen Zhao
admin_assigned
2024-05-14T08:17:50.450Z
663ab4bfade1a39663c78dff
/avatars/d9769941408ed6fb2373091e34b056c2.svg
Tuowen Zhao
user
tuowenz
6642e376ccd85d80ae1f5f75
Mark Gottscho
admin_assigned
2024-05-14T08:17:58.960Z
656d47a509d123964bf3eb49
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/fM5XouKzE9P6J48TbLV5A.jpeg
Mark Gottscho
user
mgottscho
6642e376ccd85d80ae1f5f76
David Jackson
6642e376ccd85d80ae1f5f77
Mark Luttrell
6642e376ccd85d80ae1f5f78
Manish K. Shah
6642e376ccd85d80ae1f5f79
Edison Chen
6642e376ccd85d80ae1f5f7a
Kaizhao Liang
admin_assigned
2024-05-14T08:18:26.156Z
62140dcdcf7928035e8135ad
/avatars/146a8ac3fb476e79b6503373df0f38c1.svg
Kaizhao Liang
user
kz919
6642e376ccd85d80ae1f5f7b
Swayambhoo Jain
claimed_verified
2024-05-24T09:31:47.085Z
645d7ed4f4f49de580a00fd1
/avatars/a47839f93a01a64411987f075d167370.svg
Swayambhoo Jain
user
swayambhooj
6642e376ccd85d80ae1f5f7c
Urmish Thakker
6642e376ccd85d80ae1f5f7d
Dawei Huang
admin_assigned
2024-05-14T08:19:27.846Z
65f75e3df89ec850f7ac7f01
/avatars/4ec9b4a4e42207907acbb87af17e7b1e.svg
Dawei Huang
user
daweih
6642e376ccd85d80ae1f5f7e
Sumti Jairath
6642e376ccd85d80ae1f5f7f
Kevin J. Brown
6642e376ccd85d80ae1f5f80
Kunle Olukotun
SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and
  Composition of Experts
Monolithic large language models (LLMs) like GPT-4 have paved the way for
modern generative AI applications. Training, serving, and maintaining
monolithic LLMs at scale, however, remains prohibitively expensive and
challenging. The disproportionate increase in compute-to-memory ratio of modern
AI accelerators have created a memory wall, necessitating new methods to deploy
AI. Composition of Experts (CoE) is an alternative modular approach that lowers
the cost and complexity of training and serving. However, this approach
presents two key challenges when using conventional hardware: (1) without fused
operations, smaller models have lower operational intensity, which makes high
utilization more challenging to achieve; and (2) hosting a large number of
models can be either prohibitively expensive or slow when dynamically switching
between them.
  In this paper, we describe how combining CoE, streaming dataflow, and a
three-tier memory system scales the AI memory wall. We describe Samba-CoE, a
CoE system with 150 experts and a trillion total parameters. We deploy
Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) - a
commercial dataflow accelerator architecture that has been co-designed for
enterprise inference and training applications. The chip introduces a new
three-tier memory system with on-chip distributed SRAM, on-package HBM, and
off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out
over multiple sockets. We demonstrate speedups ranging from 2x to 13x on
various benchmarks running on eight RDU sockets compared with an unfused
baseline. We show that for CoE inference deployments, the 8-socket RDU Node
reduces machine footprint by up to 19x, speeds up model switching time by 15x
to 31x, and achieves an overall speedup of 3.7x over a DGX H100 and 6.6x over a
DGX A100.
Samba-CoE, a Composition of Experts system with a three-tier memory architecture, achieves significant speedups and efficiency in deploying large language models using SambaNova's Reconfigurable Dataflow Unit.
Large Language Models Composition of Experts Samba-CoE SambaNova SN40L Reconfigurable Dataflow Unit three-tier memory system on-chip distributed SRAM on-package HBM off-package DDR DRAM inter-RDU network speedup machine footprint model switching time