title: ChipNeMo: Domain-Adapted LLMs for Chip Design
summary: ChipNeMo aims to explore the applications of large language models (LLMs) for
  industrial chip design. Instead of directly deploying off-the-shelf commercial
  or open-source LLMs, we instead adopt the following domain adaptation
  techniques: custom tokenizers, domain-adaptive continued pretraining,
  supervised fine-tuning (SFT) with domain-specific instructions, and
  domain-adapted retrieval models. We evaluate these methods on three selected
  LLM applications for chip design: an engineering assistant chatbot, EDA script
  generation, and bug summarization and analysis. Our results show that these
  domain adaptation techniques enable significant LLM performance improvements
  over general-purpose base models across the three evaluated applications,
  enabling up to 5x model size reduction with similar or better performance on a
  range of design tasks. Our findings also indicate that there's still room for
  improvement between our current results and ideal outcomes. We believe that
  further investigation of domain-adapted LLM approaches will help close this gap
  in the future.
paper_authors._id: 6542f627d0f52498c6a8b29e
paper_authors.name: Mingjie Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T13:40:57.610Z
paper_authors._id: 630a7e00450a9475bd1757e0
paper_authors.avatarUrl: /avatars/86915f782655127ecebf7f18cb1eeac8.svg
paper_authors.fullname: Mingjie Liu
paper_authors.type: user
paper_authors.user: jayl940712
paper_authors._id: 6542f627d0f52498c6a8b29f
paper_authors.name: Teo Ene
paper_authors._id: 6542f627d0f52498c6a8b2a0
paper_authors.name: Robert Kirby
paper_authors._id: 6542f627d0f52498c6a8b2a1
paper_authors.name: Chris Cheng
paper_authors._id: 6542f627d0f52498c6a8b2a2
paper_authors.name: Nathaniel Pinckney
paper_authors._id: 6542f627d0f52498c6a8b2a3
paper_authors.name: Rongjian Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T13:41:56.439Z
paper_authors._id: 642da54e91aa2e3597e871e6
paper_authors.avatarUrl: /avatars/2cf246e58ab054e11de996d571448b27.svg
paper_authors.fullname: rongjian liang
paper_authors.type: user
paper_authors.user: liangrj14
paper_authors._id: 6542f627d0f52498c6a8b2a4
paper_authors.name: Jonah Alben
paper_authors._id: 6542f627d0f52498c6a8b2a5
paper_authors.name: Himyanshu Anand
paper_authors._id: 6542f627d0f52498c6a8b2a6
paper_authors.name: Sanmitra Banerjee
paper_authors._id: 6542f627d0f52498c6a8b2a7
paper_authors.name: Ismet Bayraktaroglu
paper_authors._id: 6542f627d0f52498c6a8b2a8
paper_authors.name: Bonita Bhaskaran
paper_authors._id: 6542f627d0f52498c6a8b2a9
paper_authors.name: Bryan Catanzaro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T13:42:20.383Z
paper_authors._id: 6311021788942700629e6247
paper_authors.avatarUrl: /avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
paper_authors.fullname: Bryan Catanzaro
paper_authors.type: user
paper_authors.user: ctnzr
paper_authors._id: 6542f627d0f52498c6a8b2aa
paper_authors.name: Arjun Chaudhuri
paper_authors._id: 6542f627d0f52498c6a8b2ab
paper_authors.name: Sharon Clay
paper_authors._id: 6542f627d0f52498c6a8b2ac
paper_authors.name: Bill Dally
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T13:42:33.683Z
paper_authors._id: 638be9d21987d67b340f161d
paper_authors.avatarUrl: /avatars/a31a9bf8e200782ae2f58019de3732b2.svg
paper_authors.fullname: Bill Dally
paper_authors.type: user
paper_authors.user: wjdally
paper_authors._id: 6542f627d0f52498c6a8b2ad
paper_authors.name: Laura Dang
paper_authors._id: 6542f627d0f52498c6a8b2ae
paper_authors.name: Parikshit Deshpande
paper_authors._id: 6542f627d0f52498c6a8b2af
paper_authors.name: Siddhanth Dhodhi
paper_authors._id: 6542f627d0f52498c6a8b2b0
paper_authors.name: Sameer Halepete
paper_authors._id: 6542f627d0f52498c6a8b2b1
paper_authors.name: Eric Hill
paper_authors._id: 6542f627d0f52498c6a8b2b2
paper_authors.name: Jiashang Hu
paper_authors._id: 6542f627d0f52498c6a8b2b3
paper_authors.name: Sumit Jain
paper_authors._id: 6542f627d0f52498c6a8b2b4
paper_authors.name: Brucek Khailany
paper_authors._id: 6542f627d0f52498c6a8b2b5
paper_authors.name: Kishor Kunal
paper_authors._id: 6542f627d0f52498c6a8b2b6
paper_authors.name: Xiaowei Li
paper_authors._id: 6542f627d0f52498c6a8b2b7
paper_authors.name: Hao Liu
paper_authors._id: 6542f627d0f52498c6a8b2b8
paper_authors.name: Stuart Oberman
paper_authors._id: 6542f627d0f52498c6a8b2b9
paper_authors.name: Sujeet Omar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T13:47:58.197Z
paper_authors._id: 647760eb168cb428e00e17e4
paper_authors.avatarUrl: /avatars/2db78bce3744f97ea1694c9702c09264.svg
paper_authors.fullname: sujeet omar
paper_authors.type: user
paper_authors.user: sujeetomar
paper_authors._id: 6542f627d0f52498c6a8b2ba
paper_authors.name: Sreedhar Pratty
paper_authors._id: 6542f627d0f52498c6a8b2bb
paper_authors.name: Ambar Sarkar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T13:47:43.903Z
paper_authors._id: 64d53917620c17bfa0350e86
paper_authors.avatarUrl: /avatars/c2d725cdb8b1653cb41170a337901284.svg
paper_authors.fullname: Ambar Sarkar
paper_authors.type: user
paper_authors.user: ambarsarkar
paper_authors._id: 6542f627d0f52498c6a8b2bc
paper_authors.name: Zhengjiang Shao
paper_authors._id: 6542f627d0f52498c6a8b2bd
paper_authors.name: Hanfei Sun
paper_authors._id: 6542f627d0f52498c6a8b2be
paper_authors.name: Pratik P Suthar
paper_authors._id: 6542f627d0f52498c6a8b2bf
paper_authors.name: Varun Tej
paper_authors._id: 6542f627d0f52498c6a8b2c0
paper_authors.name: Kaizhe Xu
paper_authors._id: 6542f627d0f52498c6a8b2c1
paper_authors.name: Haoxing Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-02T13:46:12.539Z
paper_authors._id: 640218e44ad6464af9b2e5fd
paper_authors.avatarUrl: /avatars/aa530e9106e83313a1bf6366c7d74105.svg
paper_authors.fullname: Mark Ren
paper_authors.type: user
paper_authors.user: haoxingr
paper_title: ChipNeMo: Domain-Adapted LLMs for Chip Design
paper_summary: ChipNeMo aims to explore the applications of large language models (LLMs) for
  industrial chip design. Instead of directly deploying off-the-shelf commercial
  or open-source LLMs, we instead adopt the following domain adaptation
  techniques: custom tokenizers, domain-adaptive continued pretraining,
  supervised fine-tuning (SFT) with domain-specific instructions, and
  domain-adapted retrieval models. We evaluate these methods on three selected
  LLM applications for chip design: an engineering assistant chatbot, EDA script
  generation, and bug summarization and analysis. Our results show that these
  domain adaptation techniques enable significant LLM performance improvements
  over general-purpose base models across the three evaluated applications,
  enabling up to 5x model size reduction with similar or better performance on a
  range of design tasks. Our findings also indicate that there's still room for
  improvement between our current results and ideal outcomes. We believe that
  further investigation of domain-adapted LLM approaches will help close this gap
  in the future.
paper_ai_keywords: custom tokenizers domain-adaptive continued pretraining supervised fine-tuning (SFT) domain-specific instructions domain-adapted retrieval models engineering assistant chatbot EDA script generation bug summarization and analysis domain adaptation techniques