Pretraining score (bpc) does not decrease after pretraining saving and loading a Longformer model.
### System Info

Python 3.7.4

torch==1.12.0+rocm5.1.1
torchaudio==0.12.0+rocm5.1.1
torchmetrics==0.8.0
torchvision==0.13.0+rocm5.1.1
transformers==4.20.1
datasets==2.3.2

### Who can help?

@ydshieh

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am using a modified version of the [notebook](https://github.com/allenai/longformer/blob/master/scripts/convert_model_to_long.ipynb) provided in the official Longformer [repository](https://github.com/allenai/longformer).

I had to change the recommended versions in that repository in order to be able to load with a specific model the recommended Transformers version would not load properly. This caused the need for slight modifications in the code in order to make it suitable for newer Transformers and Datasets versions. Dataset handling was also modified in order to use a custom dataset. I also changed the code in order to convert the roberta-4096 to a Longformer-4096 so it can be used more easily for downstream tasks. 

However, the main body of the my code very similar to theirs.

The main problem comes after pretraining and saving the model in the following piece of code.

`
"""### Pretrain and Evaluate on masked language modeling (MLM)

The following functions pretrain and evaluate a model on MLM.
"""
def pretrain_and_evaluate(args, model, tokenizer, eval_only, model_path, logger, max_length=512, separate_documents=False, cache_dir=None):

    data_files = {}

    data_files["validation"] = args.val_datapath

    if not eval_only:
        # data_files["train"] = args.val_datapath
        data_files["train"] = args.train_datapath

        
    extension = "text"
    datasets = load_dataset(extension,
                            data_files=data_files, cache_dir=cache_dir)
    
    val_dataset = load_and_preprocess(datasets["validation"], tokenizer, max_length, separate_documents, logger)


    if eval_only:
        train_dataset = val_dataset
    else:
        logger.info(f'Loading and tokenizing training data is usually slow: {args.train_datapath}')

        train_dataset = val_dataset

        # TODO: remove this comment
        # train_dataset = load_and_preprocess(datasets["train"], tokenizer, max_length, separate_documents, logger)
    
    logger.info(f'Dataset loaded and pre-processed')

    logger.info(f'Set up Trainer...:')
    data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)
    trainer = Trainer(model=model, args=args, data_collator=data_collator,
                    train_dataset=train_dataset, eval_dataset=val_dataset)

                    #   train_dataset=train_dataset, eval_dataset=val_dataset, prediction_loss_only=True,)

    logger.info(f'Initial evaluation...:')
    eval_loss_pre = trainer.evaluate()
    eval_loss_pre_int = eval_loss_pre['eval_loss']
    logger.info(f'Initial eval bpc: {eval_loss_pre_int/math.log(2)}')
    
    bpc_results_in = {"ini_bpc": eval_loss_pre_int}

    if not eval_only:
        logger.info(f"Start pretraining from checkpoint {model_path}")
        trainer_results = trainer.train(resume_from_checkpoint=model_path)
        logger.info(f"Trainer results metrics: {trainer_results.metrics}")
        logger.info(f"Saving after pretraining in {model_path}")
        trainer.save_model(output_dir=model_path)

        logger.info(f'Mid evaluation...:')
        eval_loss_post = trainer.evaluate()
        eval_loss_post_int = eval_loss_post['eval_loss']
        logger.info(f'Eval bpc after pretraining: {eval_loss_post_int/math.log(2)}')

        bpc_results_in["end_bpc"] = eval_loss_post_int

        model_copy = LongformerForMaskedLM.from_pretrained(model_path)

        trainer_copy = Trainer(model=model_copy, args=args, data_collator=data_collator,train_dataset=train_dataset, eval_dataset=val_dataset)

        eval_loss_pre_copy = trainer_copy.evaluate()
        eval_loss_pre_copy_int = eval_loss_pre_copy['eval_loss']

        logger.info(f'Eval bpc after pretraining and loading: {eval_loss_pre_copy_int/math.log(2)}')

    return bpc_results_in
`

Where the variable model has been previously loaded as:
 `
tokenizer = LongformerTokenizerFast.from_pretrained(model_path_longformer, cache_dir=model_args.cache_dir, model_max_length=4096)
model = LongformerForMaskedLM.from_pretrained(model_path_longformer, cache_dir=model_args.cache_dir)
`

And the function is called as:

`
bpc_results = pretrain_and_evaluate(training_args, model, tokenizer, eval_only=False, model_path=model_path_longformer, logger=logger,
    cache_dir=model_args.cache_dir, max_length=model_args.max_pos, separate_documents=data_args.separate_documents)
`

As you may observe I am:

- Making and initial evaluation of the model (BPC0).
- Pretraining the model and saving it. I am pretraining one step on the validation set in order to reduce computation.
- Making an evaluation of the model (BPCa).
- Loading the pretraining model another evaluation (BPCb).

Since I am in a multi-node multi-gpu setting, I am using torch.distributed.run to launch the script. I am also setting these variables to False in the script arguments:

`  --log_on_each_node False \
  --save_on_each_node False \
`

### Expected behavior

I am trying to pre-train a Longformer on a custom text dataset starting from a RoBERTa checkpoint as indicated in the Longformer repository. From a normal execution, one would expect BPC0 > BPCa and BPCa == BPCb.

However, what I am experiencing is BPC0 (2.0194287300109863) > BPCa (2.016040325164795) and BP0 == BPCb (2.0194287300109863). Which means that the pretraining seems to be working succesfully but somehow after saving and loading the model, the weights are not being managed/saved properly. 
Hi @muxitox. If I understand correctly, the issue is about training->saving->loading->eval won't give the same result as the eval result before saving.

Could you provide a self-contained complete code snippet that can reproduce the issue? You probably don't need a full training (we can't debug if the training needs 2 days). 

- By `self-contained complete`, it means we can copy a single block of code and run it directly.
- Could you try with a much smaller model that runs on a single GPU? You can also feed the model with shorter input sequence.

Thanks a lot!


<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.<|||||>Sorry for the delay. Did the experiment you suggested. In  code_reqs.zip you can find the `convert_model_to_long_updated_github.py` script with a minimal setup to reproduce what you said in one gpu.

You can also find in requirements.txt all the packages installed in the environment and in requirements_minimal.txt the minimal requirements I think you should need to make this going.

[code_reqs.zip](https://github.com/huggingface/transformers/files/10122396/code_reqs.zip)

I have obtained the following results:

`{"ini_bpc": 1.6242934465408325, "end_bpc": 1.5721702575683594, "end_bpc_review": 1.5611544847488403, "proj_end_bpc": 1.5611544847488403, "bef_proj_end_bpc": 1.5611544847488403, "roberta_ini_bpc": 1.4125990867614746, "roberta_end_bpc": 1.0448211431503296, "roberta_end_bpc_review": 1.0905007123947144}`

Where `ini_bpc` is the bpc before pre-training the longformer, `end_bpc` is the bpc after pre-training, `end_bpc_review` is the bpc after saving and reloading the pre-trained model, `bef_proj_end_bpc` is the bpc before manually modifying the global attention as suggested in the Notebook in the original Longformer repository and  `proj_end_bpc` is the bpc after modifying the global attention. Same naming convention applies for the RoBERTa model. 

We can see that after training for 1 step on the dev split the bpc diminishes after pre-training. After re-loading, `end_bpc_review` keeps being lower even though its not exactly the same as in `end_bpc` (I think due to the data collator masking different tokens each evaluation). 

You can reproduce this executing the following line of code after laoding the installed environment:

`python convert_model_to_long_updated_github.py --output_dir issue_results  --per_device_eval_batch_size 8  --per_device_train_batch_size 2  --gradient_accumulation_steps 1 --seed 1 --learning_rate 0.00003`

The behavior we observe for this script is the expected one I think, though in my original multi-node multi-gpu set-up the problem was that end_bpc_review = ini_bpc.

In that case, this is the way I invoked the .py script (which is different than the one I shared to account for multi-node multi-gpu distribution):
```
python -m torch.distributed.run $DIST_ARGS ../../convert_model_to_long_updated.py \
  --dataset_name_or_loading_script $dataset \
  --separate_documents \
  --model_name_or_path $model \
  --seed $SEED \
  --warmup_steps 500 \
  --learning_rate $lr \
  --weight_decay "0.01" \
  --adam_epsilon "1e-6" \
  --logging_steps 500 \
  --save_strategy steps \
  --save_steps 6500 \
  --max_grad_norm "5.0" \
  --per_device_eval_batch_size 8 \
  --per_device_train_batch_size 2  \
  --gradient_accumulation_steps $GRADIENT_ACCUM \
  --evaluation_strategy steps \
  --eval_steps 500 \
  --do_train \
  --do_eval \
  --cache_dir $CACHE_DIR \
  --output_dir $OUTPUT_DIR \
  --logging_dir $LOGGING_DIR \
  --max_steps $STEPS \
  --log_on_each_node False \
  --save_on_each_node False \
```
I do not know if I set up something wrong or if indeed there is some issue in the saving process in multi-node settings.

Just for context, I work in a cluster with 2 AMD GPU Instinct™ MI50 per node, so we use torch for ROCM5.1.1.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.