Issues in visualizing a fine tuned model
BertModel finetuned for a sequence classification task does not give expected results on visualisation.
Ideally, the pretrained model should be loaded into BertForSequenceClassification, but that model does not return attentions scores for visualisation.
When loaded into BertModel (0 to 11 layers), I assume the 11th layer (right before classification layer in BertForSequenceClassification) is the right layer to check attention distribution.
But every word is attentive to every other word.
I am wondering what can be the possible reasons and how I can fix it.
Thanks.

<img width="753" alt="Screenshot 2019-07-30 at 11 19 46 AM" src="https://user-images.githubusercontent.com/25073753/62104050-08a2e600-b2bc-11e9-889a-88a6c0c9e2ea.png">

"But every word is attentive to every other word." --> I don't think that's an error, that's the general way how attention mechanism works. But definitely weights of these attentions to a particular word would vary and based on these weighted attentions and other contextual info. the downstream tasks (entailment, prediction, classification etc.) would be performed. I haven't worked on attention viz yet, but I think checking [BertViz](https://github.com/jessevig/bertviz) repo. or posting your issue over there would be more fruitful. <|||||>I don't know which framework you use for visualizing attention so I can't really help but a way to make the model output attention weights is by loading it like that:
```
model = BertModel.from_pretrained("bert-base-uncased", state_dict=model_state_dict, output_attentions=True).
```

The model will then output a tuple with the last element being the full list of attentions weights (see docstring and doc of the model).<|||||>@thomwolf I see similar results with output_attentions=True. 
The output predictions are correct but the attention scores are comparatively higher than in the case with pretrained model ('bert-base-uncased'). This makes me think, if the attention scores extracted are even correct or not for a fine tuned model.

I used [BertViz](https://github.com/jessevig/bertviz/tree/master/bertviz) as a tool for visualization.<|||||>Hi @chikubee , how does the visualization of lower layers look like? First of all, I don't think visualizing final layers is a good idea. I also tried BertVis, and I found that attention weights of higher layers are usually more uniformly distributed, like the screenshot you provided. Although I used it for Transformer encoder-decoder, but I think the phenomenon is similar. Would be interested to see your lower layer visualization.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
