adding --fp16 to run_language_modeling and increase batch size but cuda out of memory error
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 
Hi all

I am using colab ,  1 GPU , Tesla P100-PCIE-16GB

code below ran OK 

!python /content/transformers/examples/run_language_modeling.py \
    --output_dir=/content/outputs \
    --model_type=bert \
    --model_name_or_path=bert-base-cased \
    --num_train_epochs 1\
    --do_train \
    --do_eval \
    --per_gpu_train_batch_size 152\
    --train_data_file=/content/input_data/trn.txt \
    --eval_data_file=/content/input_data/val.txt \
    --evaluate_during_training \
    --learning_rate 1e-4\
    --overwrite_output_dir\
    --tokenizer_name /content/token/ \
    --block_size 64\
    --mlm





(and batch_size 152 was max num i was able to run without cuda out of memory )
then installing apex by 




%%writefile setup.sh

export CUDA_HOME=/usr/local/cuda-10.1
git clone https://github.com/NVIDIA/apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./apex

!sh setup.sh



then adding "    --fp16\"  to code but i was not able to increase batch size , even abit

do you know that ?
@thomwolf , @VictorSanh , @aaugustin , @BramVanroy , @julien-c , @LysandreJik<|||||>is it also the case with GTX 1080 , any one tried ?
<|||||>and one more thing , 
does any function in those scripts , concatenate the short lines to each other ?
in order not to be enforced to pad each line so much   <|||||>Please don't mass-tag people — thanks.<|||||>solve :
that was because i was using p100