Answers.Body: <p><a href="https://spacy.io/api/lemmatizer" rel="nofollow noreferrer">https://spacy.io/api/lemmatizer</a> just uses lookup tables and the only upstream task it relies on is POS tagging, so it should be relatively fast. For large amounts of text, <a href="https://spacy.io/usage/processing-pipelines#processing" rel="nofollow noreferrer">SpaCy recommends</a> using <a href="https://spacy.io/api/language#pipe" rel="nofollow noreferrer"><code>nlp.pipe</code></a>, which can work in batches and has built in support for multiprocessing (with the <code>n_process</code> keyword), rather than than simply <code>nlp</code>.</p>
  <p>Also, make sure you disable any pipeline elements that you don't plan to use, as they'll just waste processing time. If you're only doing lemmatization, you'll pass <code>disable=[&quot;parser&quot;, &quot;ner&quot;]</code> to the <code>nlp.pipe</code> call.</p>
  <p>Example code that takes all of the above into account is below.</p>
  <pre><code>import spacy
  nlp = spacy.load(&quot;en_core_web_sm&quot;)
  
  docs = [&quot;We've been running all day.&quot;, &quot;Let's be better.&quot;]
  
  for doc in nlp.pipe(docs, batch_size=32, n_process=3, disable=[&quot;parser&quot;, &quot;ner&quot;]):
      print([tok.lemma_ for tok in doc])
  
  # ['-PRON-', 'have', 'be', 'run', 'all', 'day', '.']
  # ['let', '-PRON-', 'be', 'well', '.']
  <span class="math-container">```</span>
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-17T15:37:05.093Z
Answers.LastActivityDate: 2020-06-17T15:37:05.093Z
Answers.UserDisplayName: primussucks
Body: <p>I am applying spacy lemmatization on my dataset, but already 20-30 mins passed and the code is still running. </p>
  
  <p>Is there anyway to make it faster? Is there any option to do this process using GPU?</p>
  
  <p>My dataset size is 20k number of rows &amp; 3 columns </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-17T14:48:14.610Z
Comments.Text: Are you using https://spacy.io/api/lemmatizer? I don't know how lemmatization works, but have you tried to parallelize your code? For example, can you split your dataset into sub-datasets and perform lemmatization on each subset on its separate CPU core or GPU?
Comments.UserDisplayName: nbro
Tags: <natural-language-processing><python><spacy>
Title: How to make spacy lemmatization process fast?
UserDisplayName: Cathrine
ThreadText: # How to make spacy lemmatization process fast?
  
  (Asked by: username_0 on 2020-06-16)
  
  I am applying spacy lemmatization on my dataset, but already 20-30 mins passed and the code is still running.
  
  Is there anyway to make it faster? Is there any option to do this process using GPU?
  
  My dataset size is 20k number of rows & 3 columns
  
  **Tags:** natural-language-processing, python, spacy
  
  > **Comments:**
  > **username_1**: Are you using https://spacy.io/api/lemmatizer? I don't know how lemmatization works, but have you tried to parallelize your code? For example, can you split your dataset into sub-datasets and perform lemmatization on each subset on its separate CPU core or GPU?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-06-17. Score: 4 | Accepted answer
  
  <https://spacy.io/api/lemmatizer> just uses lookup tables and the only upstream task it relies on is POS tagging, so it should be relatively fast. For large amounts of text, [SpaCy recommends](https://spacy.io/usage/processing-pipelines#processing) using [`nlp.pipe`](https://spacy.io/api/language#pipe), which can work in batches and has built in support for multiprocessing (with the `n_process` keyword), rather than than simply `nlp`.
  
  Also, make sure you disable any pipeline elements that you don't plan to use, as they'll just waste processing time. If you're only doing lemmatization, you'll pass `disable=["parser", "ner"]` to the `nlp.pipe` call.
  
  Example code that takes all of the above into account is below.
  
  ```
  import spacy
  nlp = spacy.load("en_core_web_sm")
  
  docs = ["We've been running all day.", "Let's be better."]
  
  for doc in nlp.pipe(docs, batch_size=32, n_process=3, disable=["parser", "ner"]):
      print([tok.lemma_ for tok in doc])
  
  # ['-PRON-', 'have', 'be', 'run', 'all', 'day', '.']
  # ['let', '-PRON-', 'be', 'well', '.']
  ```
  
  ```
  
  
  ---
  
  