A Large Encoder-Decoder Family of Foundation Models For Chemical
  Language
Large-scale pre-training methodologies for chemical language models represent
a breakthrough in cheminformatics. These methods excel in tasks such as
property prediction and molecule generation by learning contextualized
representations of input tokens through self-supervised learning on large
unlabeled corpora. Typically, this involves pre-training on unlabeled data
followed by fine-tuning on specific tasks, reducing dependence on annotated
datasets and broadening chemical language representation understanding. This
paper introduces a large encoder-decoder chemical foundation models pre-trained
on a curated dataset of 91 million SMILES samples sourced from PubChem, which
is equivalent to 4 billion of molecular tokens. The proposed foundation model
supports different complex tasks, including quantum property prediction, and
offer flexibility with two main variants (289M and 8times289M). Our
experiments across multiple benchmark datasets validate the capacity of the
proposed model in providing state-of-the-art results for different tasks. We
also provide a preliminary assessment of the compositionality of the embedding
space as a prerequisite for the reasoning tasks. We demonstrate that the
produced latent space is separable compared to the state-of-the-art with
few-shot learning capabilities.
66aa32f0a7081716091aab99
Eduardo Soares
extracted_pending
2024-07-31T12:49:53.838Z
66a2816e5cdb6fc163826d8a
https://cdn-avatars.huggingface.co/v1/production/uploads/66a2816e5cdb6fc163826d8a/RNLmcR_eVyT0OfYfb-LMM.jpeg
Eduardo Soares
user
eduardosoares99
66aa32f0a7081716091aab9a
Victor Shirasuna
extracted_confirmed
2024-08-30T13:33:00.087Z
66d1c9a6b003aeb6d16679b5
/avatars/9139af48fc764505e62a12769727d02c.svg
Victor Yukio Shirasuna
user
vshirasuna
66aa32f0a7081716091aab9b
Emilio Vital Brazil
claimed_verified
2024-08-02T07:34:56.956Z
63a202bb931d2c13ad7c721e
/avatars/a85f844d374c6f60782607e7dfca378b.svg
Emilio Ashton Vital Brazil
user
emiliobrazil
66aa32f0a7081716091aab9c
Renato Cerqueira
66aa32f0a7081716091aab9d
Dmitry Zubarev
extracted_pending
2024-07-31T12:49:53.838Z
635868fc70c0738527a8f0b8
/avatars/9d633c23661b97ea4122439a2622c8a8.svg
Dmitry Zubarev
user
dzubarev
66aa32f0a7081716091aab9e
Kristin Schmidt
A Large Encoder-Decoder Family of Foundation Models For Chemical
  Language
Large-scale pre-training methodologies for chemical language models represent
a breakthrough in cheminformatics. These methods excel in tasks such as
property prediction and molecule generation by learning contextualized
representations of input tokens through self-supervised learning on large
unlabeled corpora. Typically, this involves pre-training on unlabeled data
followed by fine-tuning on specific tasks, reducing dependence on annotated
datasets and broadening chemical language representation understanding. This
paper introduces a large encoder-decoder chemical foundation models pre-trained
on a curated dataset of 91 million SMILES samples sourced from PubChem, which
is equivalent to 4 billion of molecular tokens. The proposed foundation model
supports different complex tasks, including quantum property prediction, and
offer flexibility with two main variants (289M and 8times289M). Our
experiments across multiple benchmark datasets validate the capacity of the
proposed model in providing state-of-the-art results for different tasks. We
also provide a preliminary assessment of the compositionality of the embedding
space as a prerequisite for the reasoning tasks. We demonstrate that the
produced latent space is separable compared to the state-of-the-art with
few-shot learning capabilities.
A large-scale pre-trained encoder-decoder chemical language model achieves state-of-the-art performance across various tasks by leveraging a vast dataset of molecular structures, exhibiting strong few-shot learning capabilities.
chemical language models large-scale pre-training self-supervised learning SMILES encoder-decoder model quantum property prediction few-shot learning latent space embedding space