Tx-LLM: A Large Language Model for Therapeutics
Developing therapeutics is a lengthy and expensive process that requires the
satisfaction of many different criteria, and AI models capable of expediting
the process would be invaluable. However, the majority of current AI approaches
address only a narrowly defined set of tasks, often circumscribed within a
particular domain. To bridge this gap, we introduce Tx-LLM, a generalist large
language model (LLM) fine-tuned from PaLM-2 which encodes knowledge about
diverse therapeutic modalities. Tx-LLM is trained using a collection of 709
datasets that target 66 tasks spanning various stages of the drug discovery
pipeline. Using a single set of weights, Tx-LLM simultaneously processes a wide
variety of chemical or biological entities(small molecules, proteins, nucleic
acids, cell lines, diseases) interleaved with free-text, allowing it to predict
a broad range of associated properties, achieving competitive with
state-of-the-art (SOTA) performance on 43 out of 66 tasks and exceeding SOTA on
22. Among these, Tx-LLM is particularly powerful and exceeds best-in-class
performance on average for tasks combining molecular SMILES representations
with text such as cell line names or disease names, likely due to context
learned during pretraining. We observe evidence of positive transfer between
tasks with diverse drug types (e.g.,tasks involving small molecules and tasks
involving proteins), and we study the impact of model size, domain finetuning,
and prompting strategies on performance. We believe Tx-LLM represents an
important step towards LLMs encoding biochemical knowledge and could have a
future role as an end-to-end tool across the drug discovery development
pipeline.
6667ced234b67fd8f366a4b1
Juan Manuel Zambrano Chaves
admin_assigned
2024-06-11T10:39:08.426Z
62b3945fcbd2a402fc6f1e22
https://cdn-avatars.huggingface.co/v1/production/uploads/1658279111255-62b3945fcbd2a402fc6f1e22.jpeg
Juan Manuel Zambrano Chaves
user
jmz
6667ced234b67fd8f366a4b2
Eric Wang
6667ced234b67fd8f366a4b3
Tao Tu
admin_assigned
2024-06-11T10:39:30.013Z
64aeab30749f04f4f4c3e6d0
/avatars/68afa3b67a8d59a6108d5c3cd081cdf5.svg
Tao Tu
user
ttaotw
6667ced234b67fd8f366a4b4
Eeshit Dhaval Vaishnav
6667ced234b67fd8f366a4b5
Byron Lee
6667ced234b67fd8f366a4b6
S. Sara Mahdavi
6667ced234b67fd8f366a4b7
Christopher Semturs
6667ced234b67fd8f366a4b8
David Fleet
6667ced234b67fd8f366a4b9
Vivek Natarajan
admin_assigned
2024-06-11T10:40:43.995Z
617a6af2941993035fbaf29c
/avatars/a389671494782f06ff8502c934090cc0.svg
Vivek Natarajan
user
vivnat
6667ced234b67fd8f366a4ba
Shekoofeh Azizi
Tx-LLM: A Large Language Model for Therapeutics
Developing therapeutics is a lengthy and expensive process that requires the
satisfaction of many different criteria, and AI models capable of expediting
the process would be invaluable. However, the majority of current AI approaches
address only a narrowly defined set of tasks, often circumscribed within a
particular domain. To bridge this gap, we introduce Tx-LLM, a generalist large
language model (LLM) fine-tuned from PaLM-2 which encodes knowledge about
diverse therapeutic modalities. Tx-LLM is trained using a collection of 709
datasets that target 66 tasks spanning various stages of the drug discovery
pipeline. Using a single set of weights, Tx-LLM simultaneously processes a wide
variety of chemical or biological entities(small molecules, proteins, nucleic
acids, cell lines, diseases) interleaved with free-text, allowing it to predict
a broad range of associated properties, achieving competitive with
state-of-the-art (SOTA) performance on 43 out of 66 tasks and exceeding SOTA on
22. Among these, Tx-LLM is particularly powerful and exceeds best-in-class
performance on average for tasks combining molecular SMILES representations
with text such as cell line names or disease names, likely due to context
learned during pretraining. We observe evidence of positive transfer between
tasks with diverse drug types (e.g.,tasks involving small molecules and tasks
involving proteins), and we study the impact of model size, domain finetuning,
and prompting strategies on performance. We believe Tx-LLM represents an
important step towards LLMs encoding biochemical knowledge and could have a
future role as an end-to-end tool across the drug discovery development
pipeline.
Tx-LLM, a specialized large language model fine-tuned from PaLM-2, encodes diverse therapeutic knowledge and achieves superior performance across various drug discovery tasks by interleaving chemical data with free-text.
large language model LLM PaLM-2 therapeutics drug discovery datasets tasks chemical entities biological entities small molecules proteins nucleic acids cell lines diseases SMILES representations context learning positive transfer model size domain finetuning prompting strategies biochemical knowledge