title: Python read huge file line by line with utf-8 encoding
body: <p>I want to read some quite huge files(to be precise: the google ngram 1 word dataset) and count how many times a character occurs. Now I wrote this script:</p>
  
  <pre><code>import fileinput
  files = ['../../datasets/googlebooks-eng-all-1gram-20090715-%i.csv' % value for value in range(0,9)]
  charcounts = {}
  lastfile = ''
  for line in fileinput.input(files):
      line = line.strip()
      data = line.split('\t')
      for character in list(data[0]):
          if (not character in charcounts):
              charcounts[character] = 0
          charcounts[character] += int(data[1])
      if (fileinput.filename() is not lastfile):
          print(fileinput.filename())
          lastfile = fileinput.filename()
      if(fileinput.filelineno() % 100000 == 0):
          print(fileinput.filelineno())
  print(charcounts)
  </code></pre>
  
  <p>which works fine, until it reaches approx. line 700.000 of the first file, I then get this error:</p>
  
  <pre><code>../../datasets/googlebooks-eng-all-1gram-20090715-0.csv
  100000
  200000
  300000
  400000
  500000
  600000
  700000
  Traceback (most recent call last):
    File "charactercounter.py", line 5, in &lt;module&gt;
      for line in fileinput.input(files):
    File "C:\Python31\lib\fileinput.py", line 254, in __next__
      line = self.readline()
    File "C:\Python31\lib\fileinput.py", line 349, in readline
      self._buffer = self._file.readlines(self._bufsize)
    File "C:\Python31\lib\encodings\cp1252.py", line 23, in decode
      return codecs.charmap_decode(input,self.errors,decoding_table)[0]
  UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 7771: cha
  racter maps to &lt;undefined&gt;
  </code></pre>
  
  <p>To solve this I searched the web a bit, and came up with this code:</p>
  
  <pre><code>import fileinput
  files = ['../../datasets/googlebooks-eng-all-1gram-20090715-%i.csv' % value for value in range(0,9)]
  charcounts = {}
  lastfile = ''
  for line in fileinput.input(files,False,'',0,'r',fileinput.hook_encoded('utf-8')):
      line = line.strip()
      data = line.split('\t')
      for character in list(data[0]):
          if (not character in charcounts):
              charcounts[character] = 0
          charcounts[character] += int(data[1])
      if (fileinput.filename() is not lastfile):
          print(fileinput.filename())
          lastfile = fileinput.filename()
      if(fileinput.filelineno() % 100000 == 0):
          print(fileinput.filelineno())
  print(charcounts)
  </code></pre>
  
  <p>but the hook I now use tries to read the entire, 990MB, file into the memory at once, which kind of crashes my pc. Does anyone know how to rewrite this code so that it actually works?</p>
  
  <p>p.s: the code hasn't even run all the way yet, so I don't even know if it does what it has to do, but for that to happen I first need to fix this bug.</p>
  
  <p>Oh, and I use Python 3.2</p>