Further Pretraining of Longformer RAM Consumption
Hello everybody,

I'm interested in continued pretraining of the longformer on in-domain text. I'm using the `run_language_modeling.py` script from the examples with the following options.

!python run_language_modeling.py \
    --output_dir=$output_dir$ \
    --model_type=longformer \
    --model_name_or_path=allenai/longformer-base-4096 \
    --do_train \
    --train_data_file=$train_file$ \
    --save_steps=5000 \
    --save_total_limit=1 \
    --overwrite_output_dir \
    --num_train_epochs=1 \
    --fp16 \
    --fp16_opt_level="O1" \
    --gradient_accumulation_steps=8 \
    --per_device_train_batch_size=1 \
    --logging_steps=100 \
    --line_by_line \
    --mlm

I tried running the script on Google Colab, spell.run and kaggle-notebooks with different behaviour. While on colab and spell.run the consumption of RAM continuously increases while training, it's constant on kaggle. On colab and spell.run the script terminates whenever the limit is reached (around 26gb on spell.run). Since AllenAI provides some code for pretraining as well I tried their implementation as well with the same behaviour, however it reaches RAM limits faster. The text file I'm using is rather small with a few hundred MB in size.

Can someone explain what's going on here and how solve this issue? I'm linking  @patrickvonplaten as it was suggested by the repo. Thanks a lot for any help!

Cheers

Hi, there's been some issues with memory in our recent script versions. It is possible that on kaggle you have the very latest version of the scripts, or a way older version of the scripts where the memory issue didn't exist.

I recommend you use the latest version of the script, as the memory issue is fixed and it is the most feature-complete version.<|||||>Thanks for info and help! However, the issue persists in the current version of the script. 
Is it an issue specific to the longformer or do other models experience the same behaviour?
Do you know when the memory issues began so I can check whether older versions might work? <|||||>So I tested older versions of the script, specifically from the commits that resulted in a new version (3.0.0, 3.0.1, 3.0.2, 3.1.0). The issue persists with all of them *but* only on google colab. RAM consumption begins at around 1.88gb and after around 5 minutes of training increases continuously.
I tried the same script from the commits on kaggle and observed a very different behaviour. RAM consumption at training is constant around 5.5gb. 
When running the script on roberta the same behaviour occurs.

I'd be happy about any help or suggestions!<|||||>So the docker image that kaggle uses seemed to be a work-around. If more ressources are necessary than those provided by kaggle then one can use the image easily on GCP. Though kaggle publishes the image as well in their repo.

However by now the issues seems resolved as I switched back to colab and no problems with RAM consumption occured. I'm not sure what caused this effect but no issues for now - thus I'm closing this thread.