I am using transformers library to download longformer-base-4096 model and training on my own dataset for classification using Trainer API. So, during evaluation, all the data is categorized to negative class. I am suspecting gradient vanishing problem. If anyone can respond to how this can be handled. Below are the training arguments which are passed to Trainer class.
training_args = TrainingArguments(
output_dir=’./results’,
do_train=True,
do_eval=True,
num_train_epochs=1,
per_device_train_batch_size=1,
per_device_eval_batch_size=1,
warmup_steps=500,
weight_decay=0.01,
evaluate_during_training=True,
logging_dir=’./logs’,
)