Okay sir, first I appreciate your support.
I’m using the basic transformer with NMT for automatic grammar error correction of Arabic language.
My code and model architecture is the same as the articale 1:
The few things I have changed are to use 'bert-base-multilingual-cased' Bert tokenizer and uploaded the first dataset. 

from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual-cased')

SRC = Field(tokenize = tokenizer.tokenize, 
            init_token = '<sos>', 
            eos_token = '<eos>', 
            lower = True, 
            batch_first = True, fix_length = 100)

TRG = Field(tokenize = tokenizer.tokenize, 
            init_token = '<sos>', 
            eos_token = '<eos>', 
            lower = True, 
            batch_first = True, fix_length = 100)

train_data, valid_data = TabularDataset.splits(path='./data/',train='fluent_train_8.csv',
    validation='fluent_vali_8.csv', format='csv', fields=[('src', SRC), ('trg', TRG)], skip_header=True)

train_iterator, vali_iterator, test_iterator = BucketIterator.splits((train_data, valid_data, test_data), sort_key=lambda x: len(x.src),
     batch_size = BATCH_SIZE, device = device)

After 20 epochs as above code: (Everything worked well and expected)

Upload the second dataset:

train_data, valid_data, test_data = TabularDataset.splits(path='./data/', train='train.csv',
                                    validation='devlop.csv', test='test.csv', format='csv',
                                    fields=[('src', SRC), ('trg', TRG)], skip_header=True)

train_iterator, vali_iterator, test_iterator = BucketIterator.splits((train_data, valid_data, test_data), sort_key=lambda x: len(x.src),
     batch_size = BATCH_SIZE, device = device)


Uploaded the checkpoint:

checkpoint = torch.load('model.pt')
model.load_state_dict(checkpoint['state_dict'])
optimizer.load_state_dict(checkpoint['optimizer'])           
epoch = checkpoint['epoch'] 
chk_train_loss = checkpoint['chk_train_loss']
chk_valid_loss = checkpoint['chk_valid_loss']

Unfortunately, dataset number one is huge, but the executed code is here.