load the model before training:
from simpletransformers.question_answering import QuestionAnsweringModel
import torch
train_args = {
‘learning_rate’: 3e-5,
‘num_train_epochs’: 3, ###
‘max_seq_length’: 384,
‘doc_stride’: 384,
‘max_query_length’: 64,
‘max_answer_length’:100,
‘n_best_size’:3,
‘early_stopping_consider_epochs’: True,
‘overwrite_output_dir’: False, #####
‘reprocess_input_data’: False,
‘gradient_accumulation_steps’: 8,
‘use_early_stopping’: True,
‘evaluate_during_traing’: True,
‘save_eval_checkpoints’ : True,
‘save_model_every_epoch’: True,
‘save_steps’: 2000,
‘n_gpu’: 2, ###
‘train_batch_size’: 4,
‘dataloader_num_worker’: 8, ###
‘use_early_stopping’: True,
‘early_stopping_delta’: 0.01,
‘early_stopping_metric’: ‘eval_loss’,
‘early_stopping_metric_minimize’: True,
‘early_stopping_patience’: 3,
‘evaluate_during_training_steps’: 1000,
‘mem_len’: 1024, ### Xlnet
}
cuda_available = torch.cuda.is_available()
model = QuestionAnsweringModel(‘xlnet’, ‘xlnet-base-cased’, args=train_args, use_cuda=cuda_available)


Then, I got the following warning:

Some weights of the model checkpoint at xlnet-base-cased were not used when initializing XLNetForQuestionAnswering: [‘lm_loss.weight’, ‘lm_loss.bias’]

This IS expected if you are initializing XLNetForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
This IS NOT expected if you are initializing XLNetForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of XLNetForQuestionAnswering were not initialized from the model checkpoint at xlnet-base-cased and are newly initialized: [‘start_logits.dense.weight’, ‘start_logits.dense.bias’, ‘end_logits.dense_0.weight’, ‘end_logits.dense_0.bias’, ‘end_logits.LayerNorm.weight’, ‘end_logits.LayerNorm.bias’, ‘end_logits.dense_1.weight’, ‘end_logits.dense_1.bias’, ‘answer_class.dense_0.weight’, ‘answer_class.dense_0.bias’, ‘answer_class.dense_1.weight’]
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.



the code that you use to save the model after training:

model.train_model(train_data, output_dir="/media/bizon/DATA/QA/model_saved")

the code that you use to load the saved model:
It is from my previous working code, due to the current XLNet doesn’t work.

model = QuestionAnsweringModel(‘xlnet’, ‘/content/drive/MyDrive/transformers/apex/outputs/checkpoint-24435-epoch-3’)
=================================
The environment I am using:
Ubuntu 18.04.3 LTS
The libraries I used are:
Name: transformers
Version: 3.1.0

Name: simpletransformers
Version: 0.48.3

Name: torch
Version: 1.5.0