Variance Sources
Hi,

when I change the `--seed` argument, I get a high variance between different runs on my dataset. So I was wondering where the sources of variance might come from. I see that the seed is set (e.g. in `run_squad.py`) via:

`random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)`

But how can I find out where randomization is actually used?

I found the `RandomSampler` and replaced it with a `SequentialSampler`, but the variance remains high.

I know that `modeling.py` randomly initializes the weights, but these are overwritten by the fixed weights when loading a pre-trained BERT model, e.g. `bert-base-uncased`, correct?

Can anyone point me in any other direction where my source of variance might come from?

Thanks!
Hi Carolin,

Depending on the model you are using, not all the weights are initialized from the pre-trained models. Check the details in the [overview](https://github.com/huggingface/pytorch-pretrained-BERT#overview) section of the readme to see if it's the case for you.

Apart from weights initialization and dataset shuffling other typical source of variances are the dropout layers.

Bert fine-tuning has been reported to be a high-variance process indeed, in particular on small datasets.<|||||>Hi Thomas,

thanks for the quick reply!

I'm using `BertForMaskedLM`, so the weights should be set. But yes, I didn't think of dropout, thanks for pointing that out!
