oracat/bert-paper-classifier-arxiv

bert-paper-classifier

This model is a fine-tuned version of microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract on the dataset with titles and abstracts of papers from arXiv.

Intended uses & limitations

This model is intended to predict the arXiv taxonomy terms given the paper title (and optionally its abstract) — this is relevant for papers in mathematics, computer science, physics, etc. For example, it is likely to predict cs.CL (computation and language) as the mostl likely category for the original transformers paper.

Training procedure

The code for the model fine-tuning can be found in the respective notebook.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 128
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3

Framework versions

• Transformers 4.28.1
• Pytorch 2.0.0+cu117
• Datasets 2.11.0
• Tokenizers 0.13.3