does run_lm_finetuning.py actually use --eval_batch_size?
I'm looking through this code (thanks so much for writing it, btw) and I'm not seeing whether it actually uses eval_batch_size at all. If it doesn't, is it still performing an evaluation step to assess goodness of fit? 
No, there's no evaluation step in the example script yet. What I can recommend is using your downstream task for evaluation of the pretrained BERT. Alternatively, you could of course also add some evaluation of the LM / nextSentence loss on a validation set.<|||||>Perhaps for clarity then, that parameter should be taken out of the script? <|||||>Sure, makes sense. I created a PR. Thanks for pointing this out @bsugerman .<|||||>Fixed in master now.