AstroLLaMA: Towards Specialized Foundation Models in Astronomy
Large language models excel in many human-language tasks but often falter in
highly specialized domains like scholarly astronomy. To bridge this gap, we
introduce AstroLLaMA, a 7-billion-parameter model fine-tuned from LLaMA-2 using
over 300,000 astronomy abstracts from arXiv. Optimized for traditional causal
language modeling, AstroLLaMA achieves a 30% lower perplexity than Llama-2,
showing marked domain adaptation. Our model generates more insightful and
scientifically relevant text completions and embedding extraction than
state-of-the-arts foundation models despite having significantly fewer
parameters. AstroLLaMA serves as a robust, domain-specific model with broad
fine-tuning potential. Its public release aims to spur astronomy-focused
research, including automatic paper summarization and conversational agent
development.
650116afaf1e4021439a192a
Tuan Dung Nguyen
claimed_verified
2023-09-15T07:40:25.976Z
632aaae952ce0f7d33e3177e
https://cdn-avatars.huggingface.co/v1/production/uploads/632aaae952ce0f7d33e3177e/hIC_LLLbRU2XueHf7speV.jpeg
Josh Nguyen
user
joshnguyen
650116afaf1e4021439a192b
Yuan-Sen Ting
claimed_verified
2023-09-14T11:50:54.638Z
64f12d6e057f7e90416ce3c4
/avatars/496c09189d8c5804f88a4c484dd7c35f.svg
Yuan-Sen Ting
user
tingyuansen
650116afaf1e4021439a192c
Ioana Ciucă
claimed_verified
2023-09-14T11:51:01.122Z
6243a9a86028567dfe05727d
https://cdn-avatars.huggingface.co/v1/production/uploads/1648601509592-noauth.jpeg
Ioana Ciuca 
user
errai34
650116afaf1e4021439a192d
Charlie O'Neill
650116afaf1e4021439a192e
Ze-Chang Sun
650116afaf1e4021439a192f
Maja Jabłońska
claimed_verified
2024-01-11T11:36:07.632Z
63906dd64d918a8c3494caea
https://cdn-avatars.huggingface.co/v1/production/uploads/63906dd64d918a8c3494caea/FRvJk6-eOaLKrqpOdVm4T.png
Maya
user
mjablonska
650116afaf1e4021439a1930
Sandor Kruk
650116afaf1e4021439a1931
Ernest Perkowski
claimed_verified
2025-06-03T10:45:41.136Z
64b784b008e2452d18de95be
https://cdn-avatars.huggingface.co/v1/production/uploads/64b784b008e2452d18de95be/hfxc3NZllfpyfOSDN4EvS.jpeg
Ernest Perkowski
user
ernestp56
650116afaf1e4021439a1932
Jack Miller
650116afaf1e4021439a1933
Jason Li
claimed_verified
2023-09-25T10:52:19.942Z
64d41f821058ae59739b52d2
https://cdn-avatars.huggingface.co/v1/production/uploads/64d41f821058ae59739b52d2/48IukEahbR-W54rSq8lGq.jpeg
Jason Jingshi Li
user
Jason-lm
650116afaf1e4021439a1934
Josh Peek
650116afaf1e4021439a1935
Kartheik Iyer
claimed_verified
2023-09-14T13:50:05.596Z
64d58a90050438e3b935f134
https://cdn-avatars.huggingface.co/v1/production/uploads/64d58a90050438e3b935f134/L-inP5zk0QFJh_k87fOzm.jpeg
Kartheik Iyer
user
kiyer
650116afaf1e4021439a1936
Tomasz Różański
claimed_verified
2023-09-14T09:33:04.065Z
63597f8860e2f140f44c645b
https://cdn-avatars.huggingface.co/v1/production/uploads/63597f8860e2f140f44c645b/IvAZ9azqG3BbxH5ixmc7P.png
Tomasz Różański
user
RozanskiT
650116afaf1e4021439a1937
Pranav Khetarpal
claimed_verified
2023-09-19T15:54:10.201Z
63ac39b656fdb4fbd801504c
/avatars/6f35d73c06ba309b973b890663d04952.svg
Pranav Khetarpal
user
Pranav-khetarpal
650116afaf1e4021439a1938
Sharaf Zaman
650116afaf1e4021439a1939
David Brodrick
admin_assigned
2023-09-14T07:56:34.030Z
649c2ae43178e412366b0408
/avatars/4527f632d332438a68010bfd50a445ef.svg
David Brodrick
user
dabrodo
650116afaf1e4021439a193a
Sergio J. Rodríguez Méndez
claimed_verified
2023-09-14T09:33:07.028Z
6502cc5b8e46888d6734b6f7
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/jd4XjYedhnD83-zh77ep3.jpeg
Sergio José Rodríguez Méndez
user
srodriguez142857
650116afaf1e4021439a193b
Thang Bui
admin_assigned
2023-09-14T07:56:07.883Z
64f55058b42ea0a58182394c
/avatars/127c3bdf94e10dacfc0b9f3b0a11c745.svg
Thang Bui
user
thangbui
650116afaf1e4021439a193c
Alyssa Goodman
650116afaf1e4021439a193d
Alberto Accomazzi
650116afaf1e4021439a193e
Jill Naiman
650116afaf1e4021439a193f
Jesse Cranney
claimed_verified
2023-09-14T11:50:57.777Z
6261d939bbcbd1c34f19f4da
/avatars/ae8edfd0212fc83574b65a47dbebf8e0.svg
Jesse Cranney
user
jcranney
650116afaf1e4021439a1940
Kevin Schawinski
650116afaf1e4021439a1941
UniverseTBD
AstroLLaMA: Towards Specialized Foundation Models in Astronomy
Large language models excel in many human-language tasks but often falter in
highly specialized domains like scholarly astronomy. To bridge this gap, we
introduce AstroLLaMA, a 7-billion-parameter model fine-tuned from LLaMA-2 using
over 300,000 astronomy abstracts from arXiv. Optimized for traditional causal
language modeling, AstroLLaMA achieves a 30% lower perplexity than Llama-2,
showing marked domain adaptation. Our model generates more insightful and
scientifically relevant text completions and embedding extraction than
state-of-the-arts foundation models despite having significantly fewer
parameters. AstroLLaMA serves as a robust, domain-specific model with broad
fine-tuning potential. Its public release aims to spur astronomy-focused
research, including automatic paper summarization and conversational agent
development.
AstroLLaMA, a fine-tuned large language model using astronomy abstracts, significantly reduces perplexity and improves text completions and embeddings in specialized astronomical domains.
AstroLLaMA LLaMA-2 causal language modeling perplexity embedding extraction automatic paper summarization conversational agent development