readme: ---
  license: apache-2.0
  library_name: onnx
  language:
  - af
  - ar
  - bg
  - bn
  - bs
  - ca
  - cs
  - da
  - de
  - el
  - en
  - eo
  - es
  - et
  - eu
  - fi
  - fr
  - gl
  - he
  - hi
  - hr
  - hu
  - id
  - is
  - it
  - ja
  - ka
  - ko
  - lt
  - lv
  - mk
  - ml
  - ms
  - nl
  - 'no'
  - pl
  - pt
  - ro
  - ru
  - si
  - sk
  - sl
  - sq
  - sv
  - th
  - tr
  - uk
  - vi
  - zh
  tags:
  - sentence boundary detection
  - token classification
  - nlp
  ---
  
  
  # Model Overview
  
  This model performs text sentence boundary detection (SBD) with 49 common languages.
  
  This model segments a long, punctuated text into one or more constituent sentences.
  
  A key feature is that the model is multi-lingual and language-agnostic at inference time. 
  Therefore, language tags do not need to be used and a single batch can contain multiple languages.
  
  # Model Inputs and Outputs
  The model inputs should be **punctuated** texts.  
  
  The inputs should be packed into a batch with shape `[B, T]` , with padding being the SPE model's `<pad>` token ID.
  The `<pad>` ID is required to generate a proper attention mask.
  
  The model was trained on a maximum sequence length of 256 (subwords), and may crash or perform poorly if a longer batch is processed. 
  Optimal handling of longer sequences would require some inference-time logic (wrapping/overlapping inputs and re-combining outputs).
  
  For each input subword `t`, this model predicts the probability that `t` is the final token of a sentence (i.e., a sentence boundary).
  
  # Example Usage
  
  This model has been exported to `ONNX` (opset 17) alongside the associated `SentencePiece` tokenizer.
  
  This model is intended to be downloaded and used in the native frameworks, rather than HF's API.
  
  First, let's download and prepare the ONNX and SentencePiece models:
  
  ```python
  from sentencepiece import SentencePieceProcessor
  import onnxruntime as ort
  import numpy as np
  from huggingface_hub import hf_hub_download
  from typing import List
  
  spe_path = hf_hub_download(
      repo_id="1-800-BAD-CODE/sentence_boundary_detection_multilang", filename="spe_mixed_case_64k_49lang.model"
  )
  onnx_path = hf_hub_download(
      repo_id="1-800-BAD-CODE/sentence_boundary_detection_multilang", filename="sbd_49lang_bert_small.onnx"
  )
  
  tokenizer: SentencePieceProcessor = SentencePieceProcessor(spe_path)
  ort_session: ort.InferenceSession = ort.InferenceSession(onnx_path)
  ```
  
  Next, let's define a simple function that runs inference on one text input and prints the predictions:
  
  ```python
  def run_infer(text: str, threshold: float = 0.5):
      # Encode as IDs for the model input; add BOS/EOS tags.
      ids = tokenizer.EncodeAsIds(text)
      input_ids = np.array([[tokenizer.bos_id()] + ids + [tokenizer.eos_id()]])
      # Run inference; get probablity of each token being a sentence boundary
      outputs = ort_session.run(None, {"input_ids": input_ids})
      # Shape [B, T]
      probs = outputs[0]
      # Single input is batched; keep only first element
      probs = probs[0]
      # Trim BOS/EOS
      probs = probs[1:-1]
      # Find all positions that exceed the threshold as a sentence boundary
      break_points: List[int] = np.squeeze(np.argwhere(probs > threshold), axis=1).tolist()  # noqa
      # Add the final token to the break points, to not have leftover tokens after the loop
      if (not break_points) or (break_points[-1] != len(ids) - 1):
          break_points.append(len(ids) - 1)
      # Break tokens at boundaries, convert back to text
      print(f"Input: {text}")
      for i, break_point in enumerate(break_points):
          start = 0 if i == 0 else (break_points[i - 1] + 1)
          sub_ids = ids[start : break_point + 1]
          sub_text = tokenizer.DecodeIds(sub_ids)
          print(f"\tSentence {i}: {sub_text}")
  ```
  
  Now let's run some examples. These are all from the OpenSubtitles test set.
  
  Some interesting behavior is the English acronyms (a period is not a sufficient condition for a sentence boundary) and Thai (spaces are full stops in Thai, and this is detected automatically).
  
  These are all lower-cased to make it harder, but the model is trained with mixed-case data so true-cased inputs will work as well.
  
  ```python
  # English with a lot of acronyms
  run_infer(
      "the new d.n.a. sample has been multiplexed, and the gametes are already dividing. let's get the c.p.d. over "
      "there. dinner's at 630 p.m. see that piece on you in the l.a. times? chicago p.d. will eat him alive."
  )
  
  # Chinese
  run_infer("魔鬼兵團都死了？但是如果这让你不快乐就别做了。您就不能发个电报吗。我們都準備好了。")
  
  # Spanish
  run_infer("él es uno de aquellos. ¿tiene algo de beber? cómo el aislamiento no vale la pena.")
  
  # Thai
  run_infer(
      "พวกเขาต้องโกรธมากเลยใช่ไหม โทษทีนะลูกของเราไม่เป็นอะไรใช่ไหม ถึงเจ้าจะลากข้าไปเจ้าก็ไม่ได้อะไรอยู่ดี ผมคิดว่าจะดีกว่านะถ้าคุณไม่ออกไปไหน"
  )
  
  # Ukrainian
  run_infer(
      "розігни і зігни, будь ласка. я знаю, ваши люди храбры. было приятно, правда? для начала, тебе нужен собственный "
      "свой самолет."
  )
  
  # Polish
  run_infer(
      "szedłem tylko do. pamiętaj, nigdy się nie obawiaj żyć na krawędzi ryzyka. ćwiczę już od dwóch tygodni a byłem "
      "zabity tylko raz."
  )
  ```
  
  Expec