readme: ---
  pipeline_tag: sentence-similarity
  tags:
  - sentence-transformers
  - feature-extraction
  - sentence-similarity
  language: en
  license: apache-2.0
  datasets:
  - openwebtext
  ---
  
  
  # DeCLUTR-small
  
  ## Model description
  
  The "DeCLUTR-small" model from our paper: [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://arxiv.org/abs/2006.03659).
  
  ## Intended uses & limitations
  
  The model is intended to be used as a universal sentence encoder, similar to [Google's Universal Sentence Encoder](https://tfhub.dev/google/universal-sentence-encoder/4) or [Sentence Transformers](https://github.com/UKPLab/sentence-transformers).
  
  #### How to use
  
  Please see [our repo](https://github.com/JohnGiorgi/DeCLUTR) for full details. A simple example is shown below.
  
  ##### With [SentenceTransformers](https://www.sbert.net/)
  
  ```python
  from scipy.spatial.distance import cosine
  from sentence_transformers import SentenceTransformer
  
  # Load the model
  model = SentenceTransformer("johngiorgi/declutr-small")
  
  # Prepare some text to embed
  texts = [
      "A smiling costumed woman is holding an umbrella.",
      "A happy woman in a fairy costume holds an umbrella.",
  ]
  
  # Embed the text
  embeddings = model.encode(texts)
  
  # Compute a semantic similarity via the cosine distance
  semantic_sim = 1 - cosine(embeddings[0], embeddings[1])
  ```
  
  ##### With 🤗 Transformers
  
  ```python
  import torch
  from scipy.spatial.distance import cosine
  from transformers import AutoModel, AutoTokenizer
  
  # Load the model
  tokenizer = AutoTokenizer.from_pretrained("johngiorgi/declutr-small")
  model = AutoModel.from_pretrained("johngiorgi/declutr-small")
  
  # Prepare some text to embed
  text = [
      "A smiling costumed woman is holding an umbrella.",
      "A happy woman in a fairy costume holds an umbrella.",
  ]
  inputs = tokenizer(text, padding=True, truncation=True, return_tensors="pt")
  
  # Embed the text
  with torch.no_grad():
      sequence_output = model(**inputs)[0]
  
  # Mean pool the token-level embeddings to get sentence-level embeddings
  embeddings = torch.sum(
      sequence_output * inputs["attention_mask"].unsqueeze(-1), dim=1
  ) / torch.clamp(torch.sum(inputs["attention_mask"], dim=1, keepdims=True), min=1e-9)
  
  # Compute a semantic similarity via the cosine distance
  semantic_sim = 1 - cosine(embeddings[0], embeddings[1])
  ```
  
  ### BibTeX entry and citation info
  
  ```bibtex
  @inproceedings{giorgi-etal-2021-declutr,
  	title        = {{D}e{CLUTR}: Deep Contrastive Learning for Unsupervised Textual Representations},
  	author       = {Giorgi, John  and Nitski, Osvald  and Wang, Bo  and Bader, Gary},
  	year         = 2021,
  	month        = aug,
  	booktitle    = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  	publisher    = {Association for Computational Linguistics},
  	address      = {Online},
  	pages        = {879--895},
  	doi          = {10.18653/v1/2021.acl-long.72},
  	url          = {https://aclanthology.org/2021.acl-long.72}
  }
  
  ```