title: AstroLLaMA: Towards Specialized Foundation Models in Astronomy
summary: Large language models excel in many human-language tasks but often falter in
  highly specialized domains like scholarly astronomy. To bridge this gap, we
  introduce AstroLLaMA, a 7-billion-parameter model fine-tuned from LLaMA-2 using
  over 300,000 astronomy abstracts from arXiv. Optimized for traditional causal
  language modeling, AstroLLaMA achieves a 30% lower perplexity than Llama-2,
  showing marked domain adaptation. Our model generates more insightful and
  scientifically relevant text completions and embedding extraction than
  state-of-the-arts foundation models despite having significantly fewer
  parameters. AstroLLaMA serves as a robust, domain-specific model with broad
  fine-tuning potential. Its public release aims to spur astronomy-focused
  research, including automatic paper summarization and conversational agent
  development.
paper_authors._id: 650116afaf1e4021439a192a
paper_authors.name: Tuan Dung Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-15T07:40:25.976Z
paper_authors._id: 632aaae952ce0f7d33e3177e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632aaae952ce0f7d33e3177e/hIC_LLLbRU2XueHf7speV.jpeg
paper_authors.fullname: Josh Nguyen
paper_authors.type: user
paper_authors.user: joshnguyen
paper_authors._id: 650116afaf1e4021439a192b
paper_authors.name: Yuan-Sen Ting
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-14T11:50:54.638Z
paper_authors._id: 64f12d6e057f7e90416ce3c4
paper_authors.avatarUrl: /avatars/496c09189d8c5804f88a4c484dd7c35f.svg
paper_authors.fullname: Yuan-Sen Ting
paper_authors.type: user
paper_authors.user: tingyuansen
paper_authors._id: 650116afaf1e4021439a192c
paper_authors.name: Ioana Ciucă
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-14T11:51:01.122Z
paper_authors._id: 6243a9a86028567dfe05727d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648601509592-noauth.jpeg
paper_authors.fullname: Ioana Ciuca 
paper_authors.type: user
paper_authors.user: errai34
paper_authors._id: 650116afaf1e4021439a192d
paper_authors.name: Charlie O'Neill
paper_authors._id: 650116afaf1e4021439a192e
paper_authors.name: Ze-Chang Sun
paper_authors._id: 650116afaf1e4021439a192f
paper_authors.name: Maja Jabłońska
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-11T11:36:07.632Z
paper_authors._id: 63906dd64d918a8c3494caea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63906dd64d918a8c3494caea/FRvJk6-eOaLKrqpOdVm4T.png
paper_authors.fullname: Maya
paper_authors.type: user
paper_authors.user: mjablonska
paper_authors._id: 650116afaf1e4021439a1930
paper_authors.name: Sandor Kruk
paper_authors._id: 650116afaf1e4021439a1931
paper_authors.name: Ernest Perkowski
paper_authors._id: 650116afaf1e4021439a1932
paper_authors.name: Jack Miller
paper_authors._id: 650116afaf1e4021439a1933
paper_authors.name: Jason Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-25T10:52:19.942Z
paper_authors._id: 64d41f821058ae59739b52d2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d41f821058ae59739b52d2/48IukEahbR-W54rSq8lGq.jpeg
paper_authors.fullname: Jason Jingshi Li
paper_authors.type: user
paper_authors.user: Jason-lm
paper_authors._id: 650116afaf1e4021439a1934
paper_authors.name: Josh Peek
paper_authors._id: 650116afaf1e4021439a1935
paper_authors.name: Kartheik Iyer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-14T13:50:05.596Z
paper_authors._id: 64d58a90050438e3b935f134
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d58a90050438e3b935f134/L-inP5zk0QFJh_k87fOzm.jpeg
paper_authors.fullname: Kartheik Iyer
paper_authors.type: user
paper_authors.user: kiyer
paper_authors._id: 650116afaf1e4021439a1936
paper_authors.name: Tomasz Różański
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-14T09:33:04.065Z
paper_authors._id: 63597f8860e2f140f44c645b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63597f8860e2f140f44c645b/IvAZ9azqG3BbxH5ixmc7P.png
paper_authors.fullname: Tomasz Różański
paper_authors.type: user
paper_authors.user: RozanskiT
paper_authors._id: 650116afaf1e4021439a1937
paper_authors.name: Pranav Khetarpal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-19T15:54:10.201Z
paper_authors._id: 63ac39b656fdb4fbd801504c
paper_authors.avatarUrl: /avatars/6f35d73c06ba309b973b890663d04952.svg
paper_authors.fullname: Pranav Khetarpal
paper_authors.type: user
paper_authors.user: Pranav-khetarpal
paper_authors._id: 650116afaf1e4021439a1938
paper_authors.name: Sharaf Zaman
paper_authors._id: 650116afaf1e4021439a1939
paper_authors.name: David Brodrick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:56:34.030Z
paper_authors._id: 649c2ae43178e412366b0408
paper_authors.avatarUrl: /avatars/4527f632d332438a68010bfd50a445ef.svg
paper_authors.fullname: David Brodrick
paper_authors.type: user
paper_authors.user: dabrodo
paper_authors._id: 650116afaf1e4021439a193a
paper_authors.name: Sergio J. Rodríguez Méndez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-14T09:33:07.028Z
paper_authors._id: 6502cc5b8e46888d6734b6f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/jd4XjYedhnD83-zh77ep3.jpeg
paper_authors.fullname: Sergio José Rodríguez Méndez
paper_authors.type: user
paper_authors.user: srodriguez142857
paper_authors._id: 650116afaf1e4021439a193b
paper_authors.name: Thang Bui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:56:07.883Z
paper_authors._id: 64f55058b42ea0a58182394c
paper_authors.avatarUrl: /avatars/127c3bdf94e10dacfc0b9f3b0a11c745.svg
paper_authors.fullname: Thang Bui
paper_authors.type: user
paper_authors.user: thangbui
paper_authors._id: 650116afaf1e4021439a193c
paper_authors.name: Alyssa Goodman
paper_authors._id: 650116afaf1e4021439a193d
paper_authors.name: Alberto Accomazzi
paper_authors._id: 650116afaf1e4021439a193e
paper_authors.name: Jill Naiman
paper_authors._id: 650116afaf1e4021439a193f
paper_authors.name: Jesse Cranney
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-14T11:50:57.777Z
paper_authors._id: 6261d939bbcbd1c34f19f4da
paper_authors.avatarUrl: /avatars/ae8edfd0212fc83574b65a47dbebf8e0.svg
paper_authors.fullname: Jesse Cranney
paper_authors.type: user
paper_authors.user: jcranney
paper_authors._id: 650116afaf1e4021439a1940
paper_authors.name: Kevin Schawinski
paper_authors._id: 650116afaf1e4021439a1941
paper_authors.name: UniverseTBD
paper_title: AstroLLaMA: Towards Specialized Foundation Models in Astronomy
paper_summary: Large language models excel in many human-language tasks but often falter in
  highly specialized domains like scholarly astronomy. To bridge this gap, we
  introduce AstroLLaMA, a 7-billion-parameter model fine-tuned from LLaMA-2 using
  over 300,000 astronomy abstracts from arXiv. Optimized for traditional causal
  language modeling, AstroLLaMA achieves a 30% lower perplexity than Llama-2,
  showing marked domain adaptation. Our model generates more insightful and
  scientifically relevant text completions and embedding extraction than
  state-of-the-arts foundation models despite having significantly fewer
  parameters. AstroLLaMA serves as a robust, domain-specific model with broad
  fine-tuning potential. Its public release aims to spur astronomy-focused
  research, including automatic paper summarization and conversational agent
  development.
paper_ai_keywords: AstroLLaMA 7-billion-parameter model fine-tuned LLaMA-2 astronomy abstracts arXiv traditional causal language modeling perplexity domain adaptation text completions embedding extraction foundation models automatic paper summarization conversational agent development