Why does huggingface bert pooler hack make mixed precission training stable?
# ❓ Questions & Help

## Details

Huggigface BERT implementation has a hack to remove the pooler from optimizer.

https://github.com/huggingface/transformers/blob/b832d5bb8a6dfc5965015b828e577677eace601e/examples/run_squad.py#L927
```
# hack to remove pooler, which is not used
# thus it produce None grad that break apex
param_optimizer = [n for n in param_optimizer if 'pooler' not in n[0]]
```
We are trying to run pretraining on huggingface bert models. The code always diverges later during the training if this pooler hack is not applied. Everytime the reason is apex loss scaler becomes zero. 
After using the above hack, there is no divergence issue seen.

The pooler layer is a FFN with tanh activation
```
class BertPooler(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.activation = nn.Tanh()

    def forward(self, hidden_states):
        # We "pool" the model by simply taking the hidden state corresponding
        # to the first token.
        first_token_tensor = hidden_states[:, 0]
        pooled_output = self.dense(first_token_tensor)
        pooled_output = self.activation(pooled_output)
        return pooled_output
```
I even tried replacing the tanh acrivation with GELU and adding layer norm in the pooler layer. But the loss scaler became zero even faster.

My question is why this pooler hack solves numeric instability?


**https://stackoverflow.com/questions/60743907/why-does-huggingface-bert-pooler-hack-make-mixed-precission-training-stable**: 
Any update on this?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
