readme: # TinyBERT_L-4_H-312_v2 English Sentence Encoder
  
  This is distilled from the `bert-base-nli-stsb-mean-tokens` pre-trained model from [Sentence-Transformers](https://sbert.net/).
  
  The embedding vector is obtained by mean/average pooling of the last layer's hidden states.
  
  Update 20210325: Added the attention matrices imitation objective as in the TinyBERT paper, and the distill target has been changed from `distilbert-base-nli-stsb-mean-tokens` to `bert-base-nli-stsb-mean-tokens` (they have almost the same STSb performance).
  
  ## Model Comparison
  
  We compute cosine similarity scores of the embeddings of the sentence pair to get the spearman correlation on the STS benchmark (bigger is better):
  
  |                                      | Dev   | Test  |
  | ------------------------------------ | ----- | ----- |
  | bert-base-nli-stsb-mean-tokens       | .8704 | .8505 |
  | distilbert-base-nli-stsb-mean-tokens | .8667 | .8516 |
  | TinyBERT_L-4_H-312_v2-distill-AllNLI | .8587 | .8283 |
  | TinyBERT_L-4_H (20210325)            | .8551 | .8341 |
  