BERT tuning all parameters? 
Just a clarification question: 
when tuning bert parameters (say, for SQUAD), does it tune the parameters of the final parameter or the whole BERT model? 

In the examples scripts, it tunes the whole model.
But BERT models classes are just regular PyTorch `nn.Modules` so you can also freeze layer like you would do in any PyTorch module.