title: Tx-LLM: A Large Language Model for Therapeutics
summary: Developing therapeutics is a lengthy and expensive process that requires the
  satisfaction of many different criteria, and AI models capable of expediting
  the process would be invaluable. However, the majority of current AI approaches
  address only a narrowly defined set of tasks, often circumscribed within a
  particular domain. To bridge this gap, we introduce Tx-LLM, a generalist large
  language model (LLM) fine-tuned from PaLM-2 which encodes knowledge about
  diverse therapeutic modalities. Tx-LLM is trained using a collection of 709
  datasets that target 66 tasks spanning various stages of the drug discovery
  pipeline. Using a single set of weights, Tx-LLM simultaneously processes a wide
  variety of chemical or biological entities(small molecules, proteins, nucleic
  acids, cell lines, diseases) interleaved with free-text, allowing it to predict
  a broad range of associated properties, achieving competitive with
  state-of-the-art (SOTA) performance on 43 out of 66 tasks and exceeding SOTA on
  22. Among these, Tx-LLM is particularly powerful and exceeds best-in-class
  performance on average for tasks combining molecular SMILES representations
  with text such as cell line names or disease names, likely due to context
  learned during pretraining. We observe evidence of positive transfer between
  tasks with diverse drug types (e.g.,tasks involving small molecules and tasks
  involving proteins), and we study the impact of model size, domain finetuning,
  and prompting strategies on performance. We believe Tx-LLM represents an
  important step towards LLMs encoding biochemical knowledge and could have a
  future role as an end-to-end tool across the drug discovery development
  pipeline.
paper_authors._id: 6667ced234b67fd8f366a4b1
paper_authors.name: Juan Manuel Zambrano Chaves
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:39:08.426Z
paper_authors._id: 62b3945fcbd2a402fc6f1e22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658279111255-62b3945fcbd2a402fc6f1e22.jpeg
paper_authors.fullname: Juan Manuel Zambrano Chaves
paper_authors.type: user
paper_authors.user: jmz
paper_authors._id: 6667ced234b67fd8f366a4b2
paper_authors.name: Eric Wang
paper_authors._id: 6667ced234b67fd8f366a4b3
paper_authors.name: Tao Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:39:30.013Z
paper_authors._id: 64aeab30749f04f4f4c3e6d0
paper_authors.avatarUrl: /avatars/68afa3b67a8d59a6108d5c3cd081cdf5.svg
paper_authors.fullname: Tao Tu
paper_authors.type: user
paper_authors.user: ttaotw
paper_authors._id: 6667ced234b67fd8f366a4b4
paper_authors.name: Eeshit Dhaval Vaishnav
paper_authors._id: 6667ced234b67fd8f366a4b5
paper_authors.name: Byron Lee
paper_authors._id: 6667ced234b67fd8f366a4b6
paper_authors.name: S. Sara Mahdavi
paper_authors._id: 6667ced234b67fd8f366a4b7
paper_authors.name: Christopher Semturs
paper_authors._id: 6667ced234b67fd8f366a4b8
paper_authors.name: David Fleet
paper_authors._id: 6667ced234b67fd8f366a4b9
paper_authors.name: Vivek Natarajan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T10:40:43.995Z
paper_authors._id: 617a6af2941993035fbaf29c
paper_authors.avatarUrl: /avatars/a389671494782f06ff8502c934090cc0.svg
paper_authors.fullname: Vivek Natarajan
paper_authors.type: user
paper_authors.user: vivnat
paper_authors._id: 6667ced234b67fd8f366a4ba
paper_authors.name: Shekoofeh Azizi
paper_title: Tx-LLM: A Large Language Model for Therapeutics
paper_summary: Developing therapeutics is a lengthy and expensive process that requires the
  satisfaction of many different criteria, and AI models capable of expediting
  the process would be invaluable. However, the majority of current AI approaches
  address only a narrowly defined set of tasks, often circumscribed within a
  particular domain. To bridge this gap, we introduce Tx-LLM, a generalist large
  language model (LLM) fine-tuned from PaLM-2 which encodes knowledge about
  diverse therapeutic modalities. Tx-LLM is trained using a collection of 709
  datasets that target 66 tasks spanning various stages of the drug discovery
  pipeline. Using a single set of weights, Tx-LLM simultaneously processes a wide
  variety of chemical or biological entities(small molecules, proteins, nucleic
  acids, cell lines, diseases) interleaved with free-text, allowing it to predict
  a broad range of associated properties, achieving competitive with
  state-of-the-art (SOTA) performance on 43 out of 66 tasks and exceeding SOTA on
  22. Among these, Tx-LLM is particularly powerful and exceeds best-in-class
  performance on average for tasks combining molecular SMILES representations
  with text such as cell line names or disease names, likely due to context
  learned during pretraining. We observe evidence of positive transfer between
  tasks with diverse drug types (e.g.,tasks involving small molecules and tasks
  involving proteins), and we study the impact of model size, domain finetuning,
  and prompting strategies on performance. We believe Tx-LLM represents an
  important step towards LLMs encoding biochemical knowledge and could have a
  future role as an end-to-end tool across the drug discovery development
  pipeline.
paper_ai_keywords: large language model (LLM) PaLM-2 fine-tuned datasets chemical entities biological entities small molecules proteins nucleic acids cell lines diseases SMILES representations state-of-the-art (SOTA) positive transfer task scheduling diverse drug types model size domain finetuning prompting strategies end-to-end tool