title: How to update SpaCy vocabulary?
body: <p>I have created my own class for the text preprocessing that uses SpaCy library. I want to use the vocabulary based on the <code>self.nlp.vocab</code> attribute, however the vocabulary includes all of the tokens. How to update the vocabulary, such that it contains only the tokens which I yield in the <code>preprocess_document()</code> method?</p>
  <pre><code>class SpacyPreprocessor():
    def __init__(self):
      self.nlp = spacy.load(
        name='en_core_web_sm', 
        disable=[&quot;parser&quot;, &quot;ner&quot;]
      )
    
    def transform(self, data: Iterable) -&gt; list:
      return [
        # Join preprocessed docs
        ' '.join(self.preprocess_document(doc))
        # Apply initial sentence-level cleaning
        for doc in self.nlp.pipe(self.clean_document(text=str(x)) for x in data)
      ]
    
    @staticmethod
    def clean_word(word: str) -&gt; str:
      return re.sub(
          # Remove any unicode characters
          r&quot;_x([0-9a-fA-F]{4})_&quot;, '', word
      # Strip leading or trailing whitespaces &amp; punctuation marks
      # But leave any non-alphanumeric characters within words
      ).strip(whitespace + punctuation)
  
    @staticmethod
    def clean_document(text: str) -&gt; str:
      # Clean each word
      words = map(
          lambda s: SpacyPreprocessor.clean_word(s), 
          text.casefold().split()
      )
      # Replace multiple whitespaces with a single space
      return ' '.join(filter(lambda s: bool(s), words))
    
    @staticmethod  
    def preprocess_document(doc: Doc) -&gt; Generator[str, None, None]:
      # Tokenize document
      for token in doc:
        # Stopword removal
        if token.is_stop:
          continue
          
        # Lemmatization
        token = token.lemma_
  
        # Yield token longer than 1 character
        if len(token) &gt; 1:
          yield token
  </code></pre>