Finetuning BertModel to extract textual features for VQA shows bad results
## ❓ Questions & Help
I am trying to use Bert as a textual feature extractor for VQA.

This is the code for tokenizing question text in VQA.
```
    self.tokenizer = pytorch_transformers.BertTokenizer.from_pretrained(
            'bert-base-uncased')
    q = self.tokenizer.encode(q_text, add_special_tokens=True)
```

This code is for extracting features from Bert.
```

  self.bert = pytorch_transformers.BertModel.from_pretrained('bert-base-uncased',
                                                                   output_attentions=True)
    question_mask_cls = torch.arange(lengths[0]).to(self.device)
    lengths_cls = lengths
    q_mask_cls = question_mask_cls[None, :] < lengths_cls[:, None]
    q_mask_cls = q_mask_cls.to(torch.float32)
    question_embed_t, _, src_attn_list = self.bert(question_padded,
                                                 attention_mask=q_mask_cls)

    output, tgt_attn_list, tgt_src_attn_list = self.q_decoder(
            tgt=obj_feature_bbox_cls.permute(1, 0, 2),
                       memory=question_embed_t.permute(1, 0, 2),
                       memory_key_padding_mask=memory_key_padding_mask,
                       tgt_key_padding_mask=tgt_key_padding_mask)
```

If I free the parameters of Bert, it gives better results. But when I fine tune the whole model, the model does not seem to learn. I've tried the Adam optimizer of pytorch and AdamW provided by this repository. Both of them does not work.

```
optimizer = pytorch_transformers.optimization.AdamW(model.parameters(), lr=3e-5)
```

![image](https://user-images.githubusercontent.com/8081512/64243428-4ab9ea00-cf42-11e9-8e36-70121e0453ae.png)


The orange curve shows the model with Bert parameters freezed, while the pink and skyblue curve shows the model that trains Bert parameters.


Are there any potential issues I am missing?
<!-- A clear and concise description of the question. -->
It turned out the learning rate was too high (1e-4) using 3e-5 showed good results.