title: Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains
summary: Large Language Models (LLMs) have demonstrated remarkable proficiency in
  understanding and generating natural language. However, their capabilities wane
  in highly specialized domains underrepresented in the pretraining corpus, such
  as physical and biomedical sciences. This work explores how to repurpose
  general LLMs into effective task solvers for specialized domains. We introduce
  a novel, model-agnostic framework for learning custom input tags, which are
  parameterized as continuous vectors appended to the LLM's embedding layer, to
  condition the LLM. We design two types of input tags: domain tags are used to
  delimit specialized representations (e.g., chemical formulas) and provide
  domain-relevant context; function tags are used to represent specific functions
  (e.g., predicting molecular properties) and compress function-solving
  instructions. We develop a three-stage protocol to learn these tags using
  auxiliary data and domain knowledge. By explicitly disentangling task domains
  from task functions, our method enables zero-shot generalization to unseen
  problems through diverse combinations of the input tags. It also boosts LLM's
  performance in various specialized domains, such as predicting protein or
  chemical properties and modeling drug-target interactions, outperforming expert
  models tailored to these tasks.
paper_authors._id: 65c59478ed68e7aafdc698af
paper_authors.name: Junhong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:31:25.565Z
paper_authors._id: 6532e347b66f4bf689cf269a
paper_authors.avatarUrl: /avatars/76b5dddf80a24d3ef5c68b702280da82.svg
paper_authors.fullname: Junhong Shen
paper_authors.type: user
paper_authors.user: sjunhongs
paper_authors._id: 65c59478ed68e7aafdc698b0
paper_authors.name: Neil Tenenholtz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:31:32.779Z
paper_authors._id: 64edebfb562a2701cc286897
paper_authors.avatarUrl: /avatars/537bd273bbb5225779b2329acf82b104.svg
paper_authors.fullname: Neil Tenenholtz
paper_authors.type: user
paper_authors.user: neilatmsft
paper_authors._id: 65c59478ed68e7aafdc698b1
paper_authors.name: James Brian Hall
paper_authors._id: 65c59478ed68e7aafdc698b2
paper_authors.name: David Alvarez-Melis
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:32:06.466Z
paper_authors._id: 652ec11d0af36fdc872f7ba8
paper_authors.avatarUrl: /avatars/9e7c5e592cd76ab8ea9475f395e326d0.svg
paper_authors.fullname: David Alvarez Melis
paper_authors.type: user
paper_authors.user: daalvare
paper_authors._id: 65c59478ed68e7aafdc698b3
paper_authors.name: Nicolo Fusi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:32:12.409Z
paper_authors._id: 64f29de4f40f35cfa3e8618d
paper_authors.avatarUrl: /avatars/0aa2cf7d94716edaf585fc91a7c55dd2.svg
paper_authors.fullname: Nicolo Fusi
paper_authors.type: user
paper_authors.user: nfusi
paper_title: Tag-LLM: Repurposing General-Purpose LLMs for Specialized Domains
paper_summary: Large Language Models (LLMs) have demonstrated remarkable proficiency in
  understanding and generating natural language. However, their capabilities wane
  in highly specialized domains underrepresented in the pretraining corpus, such
  as physical and biomedical sciences. This work explores how to repurpose
  general LLMs into effective task solvers for specialized domains. We introduce
  a novel, model-agnostic framework for learning custom input tags, which are
  parameterized as continuous vectors appended to the LLM's embedding layer, to
  condition the LLM. We design two types of input tags: domain tags are used to
  delimit specialized representations (e.g., chemical formulas) and provide
  domain-relevant context; function tags are used to represent specific functions
  (e.g., predicting molecular properties) and compress function-solving
  instructions. We develop a three-stage protocol to learn these tags using
  auxiliary data and domain knowledge. By explicitly disentangling task domains
  from task functions, our method enables zero-shot generalization to unseen
  problems through diverse combinations of the input tags. It also boosts LLM's
  performance in various specialized domains, such as predicting protein or
  chemical properties and modeling drug-target interactions, outperforming expert
  models tailored to these tasks.
paper_ai_keywords: Large Language Models (LLMs) pretraining corpus parameter-efficient fine-tuning custom input tags continuous vectors embedding layer domain tags function tags auxiliary data domain knowledge zero-shot generalization protein properties chemical properties drug-target interactions expert models