How to use pytorch-transformers for transfer learning?
## ❓ Questions & Help

```python
pretrained_weights='bert-base-uncased'
tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
model = model_class.from_pretrained(pretrained_weights)

features = []
input_ids = torch.tensor([self.tokenizer.encode(phrase, add_special_tokens=True)])
with torch.no_grad():
    # Models outputs are tuples
    outputs = self.model(input_ids)
    last_hidden_states = outputs[0]
    avg_pool_hidden_states = np.average(last_hidden_states[0], axis=0))
    return avg_pool_hidden_states
```

I am working on Sentence Similarity problem, given sentence S, find similar sentences W. So I want to encode sentence S and find the closest top-K sentences from W.

I have read the documentations, but I want to confirm few things how do I do. the following:
- How do I get the second to last layer?
- Am I average pooling the last hidden states correctly?
Hey, so not exactly a direct answer to your question, but bert outright doesn't do amazing on sentence similarity. This repo here should help with your question and their paper does a great job at explaining how their method works https://github.com/UKPLab/sentence-transformers. I think you will find better results with that. Hope it helps.<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
