title: what is more efficient: applying text transformations to a corpora or df
body: <p>I have a <code>df</code> with about 100k rows with <code>texts</code> and <code>doc_id</code> as columns which I want to use for a text analysis.</p>
  
  <p>In order to perform some text transformations I generate a wordCorpus: </p>
  
  <pre><code>df %&gt;%  data.frame(.) %&gt;%  DataframeSource(.) %&gt;% Corpus(.) -&gt; wordCorpus
  </code></pre>
  
  <p>Standard transformations such as <code>tolower</code> works well:</p>
  
  <pre><code>wordCorpus&lt;- tm_map(wordCorpus, content_transformer(wordCorpus))
  </code></pre>
  
  <p>However, if I apply a bit a more complex transformation, I run out of memory (although I have 32GB RAM).</p>
  
  <pre><code>memory.limit(size=100000)
  complex &lt;- content_transformer(function(x) gsubfn("(\\d+)(-| - | bis | jusqu'à )(\\d+)", ~ paste(seq(..1, ..3), collapse = " "), x))
  wordCorpus&lt;- tm_map(wordCorpus,complex)
  </code></pre>
  
  <p>Would it be more efficient to perform the transformations on the <code>df</code> itself?</p>
  
  <p>something like: <code>df %&gt;% group_by(doc_id) %&gt;% mutate(texts=gsubfn("(\\d+)(-| - | bis | jusqu'à )(\\d+)", ~ paste(seq(..1, ..3), collapse = " "), .)</code></p>
  
  <p>Or is there a better way to get the transformations done with less memory usage?</p>