readme: ---
  language: 
    - en
  license: mit
  datasets:
  - pile
  metrics:
  - nDCG@10
  - MRR
  ---
  
  # Carptriever-1
  
  
  ## Model description
  
  Carptriever-1 is a `bert-large-uncased` retrieval model trained with contrastive learning via a momentum contrastive (MoCo) mechanism following the work of G. Izacard et al. in ["Contriever: Unsupervised Dense Information Retrieval with Contrastive Learning"](https://arxiv.org/abs/2112.09118).
  
  
  ## How to use
  
  ```python
  from transformers import AutoTokenizer, AutoModel
  
  def mean_pooling(token_embeddings, mask):
      token_embeddings = token_embeddings.masked_fill(~mask[..., None].bool(), 0.)
      sentence_embeddings = token_embeddings.sum(dim=1) / mask.sum(dim=1)[..., None]
      return sentence_embeddings
  
  # Remove pooling layer
  model = AutoModel.from_pretrained("CarperAI/carptriever-1", add_pooling_layer=False)
  tokenizer = AutoTokenizer.from_pretrained("CarperAI/carptriever-1")
  
  sentences = [
      "Where was Marie Curie born?",  # Query
      "Maria Sklodowska, later known as Marie Curie, was born on November 7, 1867.",
      "Born in Paris on 15 May 1859, Pierre Curie was the son of Eugène Curie, a doctor of French Catholic origin from Alsace."
  ]
  
  # Apply tokenizer
  inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
  
  # Encode sentences 
  outputs = model(**inputs)
  embeddings = mean_pooling(outputs[0], inputs['attention_mask'])
  
  # Compute dot-product scores between the query and sentence embeddings
  query_embedding, sentence_embeddings = embeddings[0], embeddings[1:]
  scores = (query_embedding @ sentence_embeddings.transpose(0, 1)).cpu().tolist()
  
  sentence_score_pairs = sorted(zip(sentences[1:], scores), reverse=True)
  print(f"Query: {sentences[0]}")
  for sentence, score in sentence_score_pairs:
      print(f"\nSentence: {sentence}\nScore: {score:.4f}")
  ```
  
  
  ## Training data
  
  Carptriever-1 is pre-trained on a de-duplicated subset of [The Pile](https://pile.eleuther.ai/), a large and diverse dataset created by EleutherAI for language model training. This subset was created through a [Minhash LSH](http://ekzhu.com/datasketch/lsh.html) process using a threshold of `0.87`.
  
  
  ## Training procedure
  
  The model was trained on 32 40GB A100 for approximately 100 hours with the following configurations:
  
  - Base model: 
      - `bert-large-uncased`
  - Optimizer settings:
      - `optimizer  = AdamW`
      - `lr         = 1e-5`
      - `schedule   = linear`
      - `warmup     = 20,000 steps`
      - `batch size = 2048`
      - `training steps = 150,000`
  - MoCo settings: 
      - `queue size  = 8192`
      - `momentum    = 0.999`
      - `temperature = 0.05` 
  
  
  ## Evaluation results
  
  #### [BEIR: Benchmarking IR](https://github.com/beir-cellar/beir)
  
  We report the following BEIR scores as measured in normalized discounted cumulative gain (nDCG@10):
  
  |   Model       | Avg   | MSMARCO | TREC-Covid | NFCorpus | NaturalQuestions | HotpotQA | FiQA | ArguAna | Tóuche-2020 | Quora | CQAdupstack | DBPedia | Scidocs | Fever | Climate-fever | Scifact  |
  |---------------|-------|---------|------------|----------|------------------|----------|------|---------|-------------|-------|-------------|---------|---------|-------|---------------|----------|
  | Contriever*   | 35.97 |    20.6 |       27.4 |     31.7 |             25.4 |     48.1 | 24.5 |    37.9 |        19.3 |  83.5 |       28.40 |    29.2 |    14.9 | 68.20 |          15.5 |    64.9  |
  | Carptriever-1 | 34.54 |   18.83 |   **52.2** |     28.5 |             21.1 |     39.4 | 23.2 |    31.7 |        15.2 |  81.3 |       26.88 |    25.4 |    14.2 | 57.36 |      **17.9** |    64.9  |
  
  \* Results are taken from the Contriever [GitHub repository](https://github.com/facebookresearch/contriever).
  
  Note that degradation in performance, relative to the Contriever model, was expected given the much broader diversity of our training dataset. We plan on addressing this in future updates with architectural improvements and view Carptriever-1 as our first iteration in the exploratory phase towards better language-embedding models.
  
  
  #### [CodeSearchNet Challenge Evaluating the State of Semantic Code Search](https://arxiv.org/pdf/1909.09436.pdf)
  
  We provide results on the CodeSearchNet benchmark, measured in Mean Reciprocal Rank (MRR), following the code search procedure outlined in Section 3.3 of Neelakantan et al.'s ["Text and Code Embeddings by Contrastive Pre-Training"](https://arxiv.org/pdf/2201.10005.pdf).
  
  `Candidate Size = 1,000`
  
  | Model           | Avg   | Python | Go    | Ruby  | PHP   | Java  | JS    |
  |-----------------|-------|--------|-------|-------|-------|-------|-------|
  | Carptriever-1   | 60.24 |  65.85 | 63.29 |  62.1 |  59.1 | 55.52 | 55.55 |
  | Contriever      | 49.39 |  54.81 |  58.9 | 55.19 | 38.46 | 44.89 | 44.09 |
  
  
  `Candidate Size = 10,000`
  
  | Model.          | Avg   | Python | Go    | Ruby  | PHP   | Java  | JS    |
  |-----------------|-------|--------|-------|-------|-------|-------|-------|
  | Carptriever-1   | 48.59 |  55.98 | 43.18 | 56.06 | 45.