readme: ---
  license: apache-2.0
  language:
  - ar
  - bs
  - cs
  - de
  - en
  - es
  - et
  - fi
  - fr
  - hi
  - is
  - it
  - ja
  - lt
  - lv
  - pl
  - pt
  - ro
  - ru
  - uk
  - tr
  - zh
  ---
  
  # Model Description
  This model accepts as input un-punctuated, lower-case text and performs the following analytics:
  * Punctuation restoration
  * True-casing
  * Sentence-boundary detection.
  
  Quick example of input/output:
  
  ```
  Input: hello world what's up
  Outputs: 
    Hello, World.
    What's up?
  ```
  (note that, technically, 'world' is a proper noun here and should be capitalized)
  
  This model was trained in "one-pass" mode, meaning that it encodes the input text once and performs all three analytics in parallel. This is a trade-off of throughput for discarding the conditional probabilities of each analytic; e.g., true-casing should be a function of sentence boundaries, but they are predicted jointly by this model.
  
  For each subword in the input sequence, this model predicts the following:
  
  * `N` pre-punctuation tokens, where `N` is the number of characters in the subword, to insert before each character
  * `N` post-punctuation tokens, where `N` is the number of characters in the subword, to insert after each character
  * `N` capitalization predictions, where `N` is the number of characters in the subword, whether to capitalize each letter
  * One sentence boundary prediction, whether this subword is the end of a sentence (if a full stop is predicted after the final character of this subword)
  
  The "pre-punctuation" tokens are predicted before every character to, in practice, predict inverted question marks preceding Spanish questions.
  
  The "post-punctuation" tokens are predicted after every character to predict typical punctuation, e.g., "Hello, world.", as well as within acronyms, e.g., "U.S.", and insert punctuation into continuous-script languages such as Chinese and Japanese in a language-agnostic manner.
  
  True-casing is predicted on a character level to account for all types of words such as "Hello", "AMC", "U.S.", "McDonald", etc.
  
  Sentence boundaries are predicted at the subword level, rather than the character level, presuming that tokenizers rarely, if ever, produce tokens which cross the boundary between two sentences. Sentence boundary predictions are applied only if a full-stop token is predicted (or present in the training example).
  
  ## Usage
  
  This model is driven by code that lives in a personal branch of NeMo. This branch must be installed to run this model.
  
  If you have NeMo and the Github CLI installed, the code can be pulled from the corresponding pull request:
  
  ```shell
  $ cd /path/to/nemo/
  $ gh pr checkout 4637
  ```
  
  Or to pull the full codebase into a separate repo:
  
  ```shell
  $ git clone -b punct_cap_seg https://github.com/1-800-BAD-CODE/nemo
  $ cd nemo
  $ ./reinstall.sh
  ```
  
  The model provides an `infer` method which accepts as input un-punctuated, lower-cased text and returns as output for each input sentence a list of new sentences, segmented on sentence boundaries, with punctuation inserted and true-casing applied.
  
  Inference is language-agnostic, and multiple language can appear in the same batch (though, sentences in multiple languages should not be concatenated).
  
  
  ### Example Input
  
  ```
  from typing import List
  
  import nemo.collections.nlp as nemo_nlp
  
  
  hf_model_name = "1-800-BAD-CODE/punct_cap_seg_22lang1pass_xlmr"
  hf_model = nemo_nlp.models.PunctCapSegModel.from_pretrained(hf_model_name, map_location="cpu")
  
  # Supported languages is for reference only. Inference is language-agnostic.
  print(f"Model's supported languages: {hf_model.cfg.supported_languages}")
  
  # Define function for pretty-printing inputs and outputs. Each input is a raw string; outputs are one or more segmented sentences. This 
  # function expects a batch.
  def print_inputs_outputs(inputs: List[str], outputs: List[List[str]]):
      for next_input, next_outputs in zip(inputs, outputs):
          print(f"Input: {next_input}")
          print(f"Outputs: ")
          for output_text in next_outputs:
              print(f"  {output_text}")
          print()
  
  
  # Input is a list of sentences. Each sentence gets punctuated, true-cased, and segmented into one or more sentences
  output_texts: List[List[str]]
  
  # Run a similar set of sentences in each supported language (sentences were translated from English with NMT; they might not be right).
  input_texts = [
      # en
      "i saw you at the park yesterday what were you doing there",
      # ar
      "رأيتك في الحديقة أمس ماذا كنت تفعل هناك",
      # bs
      "video sam te juče u parku šta si tamo radio",
      # cs
      "viděl jsem tě včera v parku co jsi tam dělal",
      # de
      "ich habe dich gestern im park gesehen was hast du dort gemacht",
      # et
      "na nägin sind eile pargis mida sa seal tegid",
      # es
      "te vi en el parque ayer que estabas haciendo ahi",
      # fi
      "näin sinut eilen puistossa mitä teit siellä",
      # fr
      "je t'ai vu au parc hier qu'est ce que tu faisais là",
      # hi
      "मैंने तुम्हें कल पार्क में देखा था तुम वहाँ क्या कर रहे थे",
  