ChipNeMo: Domain-Adapted LLMs for Chip Design
ChipNeMo aims to explore the applications of large language models (LLMs) for
industrial chip design. Instead of directly deploying off-the-shelf commercial
or open-source LLMs, we instead adopt the following domain adaptation
techniques: custom tokenizers, domain-adaptive continued pretraining,
supervised fine-tuning (SFT) with domain-specific instructions, and
domain-adapted retrieval models. We evaluate these methods on three selected
LLM applications for chip design: an engineering assistant chatbot, EDA script
generation, and bug summarization and analysis. Our results show that these
domain adaptation techniques enable significant LLM performance improvements
over general-purpose base models across the three evaluated applications,
enabling up to 5x model size reduction with similar or better performance on a
range of design tasks. Our findings also indicate that there's still room for
improvement between our current results and ideal outcomes. We believe that
further investigation of domain-adapted LLM approaches will help close this gap
in the future.
6542f627d0f52498c6a8b29e
Mingjie Liu
admin_assigned
2023-11-02T13:40:57.610Z
630a7e00450a9475bd1757e0
/avatars/93bf8e3d6a24e93118572d1a77266cbd.svg
Mingjie Liu
user
jayl940712
6542f627d0f52498c6a8b29f
Teo Ene
6542f627d0f52498c6a8b2a0
Robert Kirby
6542f627d0f52498c6a8b2a1
Chris Cheng
6542f627d0f52498c6a8b2a2
Nathaniel Pinckney
6542f627d0f52498c6a8b2a3
Rongjian Liang
admin_assigned
2023-11-02T13:41:56.439Z
642da54e91aa2e3597e871e6
/avatars/2cf246e58ab054e11de996d571448b27.svg
rongjian liang
user
liangrj14
6542f627d0f52498c6a8b2a4
Jonah Alben
6542f627d0f52498c6a8b2a5
Himyanshu Anand
6542f627d0f52498c6a8b2a6
Sanmitra Banerjee
6542f627d0f52498c6a8b2a7
Ismet Bayraktaroglu
6542f627d0f52498c6a8b2a8
Bonita Bhaskaran
6542f627d0f52498c6a8b2a9
Bryan Catanzaro
admin_assigned
2023-11-02T13:42:20.383Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
6542f627d0f52498c6a8b2aa
Arjun Chaudhuri
6542f627d0f52498c6a8b2ab
Sharon Clay
6542f627d0f52498c6a8b2ac
Bill Dally
admin_assigned
2023-11-02T13:42:33.683Z
638be9d21987d67b340f161d
/avatars/a31a9bf8e200782ae2f58019de3732b2.svg
Bill Dally
user
wjdally
6542f627d0f52498c6a8b2ad
Laura Dang
6542f627d0f52498c6a8b2ae
Parikshit Deshpande
6542f627d0f52498c6a8b2af
Siddhanth Dhodhi
6542f627d0f52498c6a8b2b0
Sameer Halepete
6542f627d0f52498c6a8b2b1
Eric Hill
6542f627d0f52498c6a8b2b2
Jiashang Hu
6542f627d0f52498c6a8b2b3
Sumit Jain
6542f627d0f52498c6a8b2b4
Brucek Khailany
6542f627d0f52498c6a8b2b5
Kishor Kunal
6542f627d0f52498c6a8b2b6
Xiaowei Li
6542f627d0f52498c6a8b2b7
Hao Liu
6542f627d0f52498c6a8b2b8
Stuart Oberman
6542f627d0f52498c6a8b2b9
Sujeet Omar
admin_assigned
2023-11-02T13:47:58.197Z
647760eb168cb428e00e17e4
/avatars/2db78bce3744f97ea1694c9702c09264.svg
sujeet omar
user
sujeetomar
6542f627d0f52498c6a8b2ba
Sreedhar Pratty
6542f627d0f52498c6a8b2bb
Ambar Sarkar
admin_assigned
2023-11-02T13:47:43.903Z
64d53917620c17bfa0350e86
/avatars/c2d725cdb8b1653cb41170a337901284.svg
Ambar Sarkar
user
ambarsarkar
6542f627d0f52498c6a8b2bc
Zhengjiang Shao
6542f627d0f52498c6a8b2bd
Hanfei Sun
6542f627d0f52498c6a8b2be
Pratik P Suthar
6542f627d0f52498c6a8b2bf
Varun Tej
6542f627d0f52498c6a8b2c0
Kaizhe Xu
6542f627d0f52498c6a8b2c1
Haoxing Ren
admin_assigned
2023-11-02T13:46:12.539Z
640218e44ad6464af9b2e5fd
/avatars/aa530e9106e83313a1bf6366c7d74105.svg
Mark Ren
user
haoxingr
ChipNeMo: Domain-Adapted LLMs for Chip Design
ChipNeMo aims to explore the applications of large language models (LLMs) for
industrial chip design. Instead of directly deploying off-the-shelf commercial
or open-source LLMs, we instead adopt the following domain adaptation
techniques: custom tokenizers, domain-adaptive continued pretraining,
supervised fine-tuning (SFT) with domain-specific instructions, and
domain-adapted retrieval models. We evaluate these methods on three selected
LLM applications for chip design: an engineering assistant chatbot, EDA script
generation, and bug summarization and analysis. Our results show that these
domain adaptation techniques enable significant LLM performance improvements
over general-purpose base models across the three evaluated applications,
enabling up to 5x model size reduction with similar or better performance on a
range of design tasks. Our findings also indicate that there's still room for
improvement between our current results and ideal outcomes. We believe that
further investigation of domain-adapted LLM approaches will help close this gap
in the future.
Custom tokenizers, continued pretraining, supervised fine-tuning, and retrieval models improve large language models for industrial chip design tasks, including engineering assistance, EDA script generation, and bug analysis.
custom tokenizers domain-adaptive continued pretraining supervised fine-tuning domain-adapted retrieval models