readme: ---
  language:
    - ar
    - de
    - en
    - es
    - fr
    - it
    - ja
    - ko
    - nl
    - pl
    - pt
    - ru
    - th
    - tr
    - zh
  tags:
    - bert
    - sentence_embedding
    - multilingual
    - google
    - sentence-similarity
    - labse
  license: apache-2.0
  datasets:
    - CommonCrawl
    - Wikipedia
  ---
  
  # LaBSE
  
  ## Model description
  
  Smaller Language-agnostic BERT Sentence Encoder (LaBSE) is a BERT-based model distilled from the [original LaBSE model](https://huggingface.co/setu4993/LaBSE) to 15 languages (from the original 109 languages) using the techniques described in the paper ['Load What You Need: Smaller Versions of Multilingual BERT'](https://arxiv.org/abs/2010.05609) by [Ukjae Jeong](https://github.com/jeongukjae/).
  
  - Model: [HuggingFace's model hub](https://huggingface.co/setu4993/smaller-LaBSE).
  - Original model: [TensorFlow Hub](https://tfhub.dev/jeongukjae/smaller_LaBSE_15lang/1).
  - Distillation source: [GitHub](https://github.com/jeongukjae/smaller-labse).
  - Conversion from TensorFlow to PyTorch: [GitHub](https://github.com/setu4993/convert-labse-tf-pt).
  
  ## Usage
  
  Using the model:
  
  ```python
  import torch
  from transformers import BertModel, BertTokenizerFast
  
  
  tokenizer = BertTokenizerFast.from_pretrained("setu4993/smaller-LaBSE")
  model = BertModel.from_pretrained("setu4993/smaller-LaBSE")
  model = model.eval()
  
  english_sentences = [
      "dog",
      "Puppies are nice.",
      "I enjoy taking long walks along the beach with my dog.",
  ]
  english_inputs = tokenizer(english_sentences, return_tensors="pt", padding=True)
  
  with torch.no_grad():
      english_outputs = model(**english_inputs)
  ```
  
  To get the sentence embeddings, use the pooler output:
  
  ```python
  english_embeddings = english_outputs.pooler_output
  ```
  
  Output for other languages:
  
  ```python
  italian_sentences = [
      "cane",
      "I cuccioli sono carini.",
      "Mi piace fare lunghe passeggiate lungo la spiaggia con il mio cane.",
  ]
  japanese_sentences = ["犬", "子犬はいいです", "私は犬と一緒にビーチを散歩するのが好きです"]
  italian_inputs = tokenizer(italian_sentences, return_tensors="pt", padding=True)
  japanese_inputs = tokenizer(japanese_sentences, return_tensors="pt", padding=True)
  
  with torch.no_grad():
      italian_outputs = model(**italian_inputs)
      japanese_outputs = model(**japanese_inputs)
  
  italian_embeddings = italian_outputs.pooler_output
  japanese_embeddings = japanese_outputs.pooler_output
  ```
  
  For similarity between sentences, an L2-norm is recommended before calculating the similarity:
  
  ```python
  import torch.nn.functional as F
  
  
  def similarity(embeddings_1, embeddings_2):
      normalized_embeddings_1 = F.normalize(embeddings_1, p=2)
      normalized_embeddings_2 = F.normalize(embeddings_2, p=2)
      return torch.matmul(
          normalized_embeddings_1, normalized_embeddings_2.transpose(0, 1)
      )
  
  
  print(similarity(english_embeddings, italian_embeddings))
  print(similarity(english_embeddings, japanese_embeddings))
  print(similarity(italian_embeddings, japanese_embeddings))
  ```
  
  ## Details
  
  Details about data, training, evaluation and performance metrics are available in the [original paper](https://arxiv.org/abs/2007.01852).
  
  ### BibTeX entry and citation info
  
  ```bibtex
  @misc{feng2020languageagnostic,
        title={Language-agnostic BERT Sentence Embedding},
        author={Fangxiaoyu Feng and Yinfei Yang and Daniel Cer and Naveen Arivazhagan and Wei Wang},
        year={2020},
        eprint={2007.01852},
        archivePrefix={arXiv},
        primaryClass={cs.CL}
  }
  ```
  