title: How to process a large list in Scala?
body: <p>I used to handle stats stuff in Python. For example, a large file contains tens of millions of ids:</p>
  
  <pre><code>$ cat report_ids | head
  3788065
  7950319
  140494477
  182851142
  120757318
  160033281
  78087029
  42591118
  104363873
  212143796
  ...
  </code></pre>
  
  <p>In IPython, the following lines works well always.</p>
  
  <pre><code>In [1]: lines = [line.strip() for  line in open('./report_ids').readlines()]
  
  In [2]: from collections import Counter
  
  In [3]: d = Counter(lines)
  
  In [4]: d[lines[0]]
  Out[4]: 9
  </code></pre>
  
  <p>When I try the same in Scala, an out of memory errors occurs.</p>
  
  <pre><code>val lines = scala.io.Source.fromFile("./report_ids").getLines.toList
  lines: List[String] = List(3788065, 7950319, 140494477, 182851142, 120757318, 160033281, 78087029, 42591118, 104363873, 212143796, 175644298, 112703123, 213308679, 109649718, 11947300, 214660563, 83402867, 162877289, 83030111, 78231639, 45129180, 11635655, 34778452, 46604760, 142519099, 213261965, 137812002, 167057636, 119258917, 212722777, 177979907, 13754217, 156769524, 40682536, 202195379, 91879046, 22766751, 6656279, 11972540, 76929862, 91616020, 110579570, 143849021, 27239477, 65146692, 142968764, 153891284, 182405787, 153038108, 50714639, 113386401, 96657813, 75908413, 32215626, 175000692, 154337083, 113754207, 165109267, 3788065, 42285876, 171004203, 109802388, 92956305, 46690091, 103638776, 15141632, 110579570, 120984867, 183167775, 86841540, 60465849, 27239477, 91760184, 213464...
  
  scala&gt; val g = lines.groupBy(e =&gt; e).mapValues(x =&gt; x.length)
  java.lang.OutOfMemoryError: GC overhead limit exceeded
      at scala.collection.immutable.HashMap$HashTrieMap.updated0(HashMap.scala:328)
      at scala.collection.immutable.HashMap$HashTrieMap.updated0(HashMap.scala:326)
      at scala.collection.immutable.HashMap.$plus(HashMap.scala:57)
      at scala.collection.immutable.HashMap.$plus(HashMap.scala:36)
      at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28)
      at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24)
      at scala.collection.TraversableLike$$anonfun$groupBy$3.apply(TraversableLike.scala:334)
      at scala.collection.TraversableLike$$anonfun$groupBy$3.apply(TraversableLike.scala:333)
      at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
      at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
      at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
      at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
      at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
      at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
      at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
      at scala.collection.TraversableLike$class.groupBy(TraversableLike.scala:333)
      at scala.collection.AbstractTraversable.groupBy(Traversable.scala:105)
      at .&lt;init&gt;(&lt;console&gt;:8)
      at .&lt;clinit&gt;(&lt;console&gt;)
      at .&lt;init&gt;(&lt;console&gt;:7)
      at .&lt;clinit&gt;(&lt;console&gt;)
      at $print(&lt;console&gt;)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:734)
      at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:983)
      at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:604)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:568)
      at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:760)
  </code></pre>
  
  <p>Then I tried the <strong>lazy</strong> method in Scala, its still not working.</p>
  
  <pre><code>scala&gt; lazy val lines = scala.io.Source.fromFile("./report_ids").getLines.toList
  lines: List[String] = &lt;lazy&gt;
  
  scala&gt; val g = lines.groupBy(e =&gt; e).mapValues(x =&gt; x.length)
  java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOfRange(Arrays.java:2694)
      at java.lang.String.&lt;init&gt;(String.java:203)
      at java.io.BufferedReader.readLine(BufferedReader.java:349)
      at java.io.BufferedReader.readLine(BufferedReader.java:382)
      at scala.io.BufferedSource$BufferedLineIterator.hasNext(BufferedSource.scala:67)
      at scala.collection.Iterator$class.foreach(Iterator.scala:727)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
      at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
      at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:176)
      at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:45)
      at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
      at scala.collection.AbstractIterator.to(Iterator.scala:1157)
      at scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:257)
      at scala.collection.AbstractIterator.toList(Iterator.scala:1157)
      at .lines$lzycompute(&lt;console&gt;:7)
      at .lines(&lt;console&gt;:7)
      at .&lt;init&gt;(&lt;console&gt;:8)
      at .&lt;clinit&gt;(&lt;console&gt;)
      at .&lt;init&gt;(&lt;console&gt;:7)
      at .&lt;clinit&gt;(&lt;console&gt;)
      at $print(&lt;console&gt;)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:734)
      at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:983)
      at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:604)
      at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:568)
      at scala.tools.nsc.interpreter.ILoop.reallyInterpret$1(ILoop.scala:760)
      at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:805)
  </code></pre>
  
  <p>So how can I finished the group by work in Scala as the way I did in Python? Thanks.</p>