Tokenizing unicode using nltk
I have textfiles that use utf-8 encoding that contain characters like 'ö', 'ü', etc. I would like to parse the text form these files, but I can't get the tokenizer to work properly. If I use standard nltk tokenizer:

    f = open('C:\Python26\text.txt', 'r') # text = 'müsli pöök rääk'
    text = f.read()
    f.close
    items = text.decode('utf8')
    a = nltk.word_tokenize(items)

---------------------------------
Output: [u'\ufeff', u'm', u'\xfc', u'sli', u'p', u'\xf6', u'\xf6', u'k', u'r', u'\xe4', u'\xe4', u'k']


Punkt tokenizer seems to do better:

    f = open('C:\Python26\text.txt', 'r') # text = 'müsli pöök rääk'
    text = f.read()
    f.close
    items = text.decode('utf8')
    a = PunktWordTokenizer().tokenize(items)

 
------------------------------
output: [u'\ufeffm\xfcsli', u'p\xf6\xf6k', u'r\xe4\xe4k']

There is still '\ufeff' before the first token that i can't figure out (not that I can't remove it). What am I doing wrong? Help greatly appreciated.
Tokenizing unicode using nltk
===

I have textfiles that use utf-8 encoding that contain characters like 'ö', 'ü', etc. I would like to parse the text form these files, but I can't get the tokenizer to work properly. If I use standard nltk tokenizer:

    f = open('C:\Python26\text.txt', 'r') # text = 'müsli pöök rääk'
    text = f.read()
    f.close
    items = text.decode('utf8')
    a = nltk.word_tokenize(items)

---------------------------------
Output: [u'\ufeff', u'm', u'\xfc', u'sli', u'p', u'\xf6', u'\xf6', u'k', u'r', u'\xe4', u'\xe4', u'k']


Punkt tokenizer seems to do better:

    f = open('C:\Python26\text.txt', 'r') # text = 'müsli pöök rääk'
    text = f.read()
    f.close
    items = text.decode('utf8')
    a = PunktWordTokenizer().tokenize(items)

 
------------------------------
output: [u'\ufeffm\xfcsli', u'p\xf6\xf6k', u'r\xe4\xe4k']

There is still '\ufeff' before the first token that i can't figure out (not that I can't remove it). What am I doing wrong? Help greatly appreciated.