Question: I have a file which is mostly UTF-8, but some Windows-1252 characters have also found their way in.

I created a table to map from the Windows-1252 (cp1252) characters to their Unicode counterparts, and would like to use it to fix the mis-encoded characters, e.g.

```
cp1252_to_unicode = {
    "\x85": u'\u2026', # …
    "\x91": u'\u2018', # ‘
    "\x92": u'\u2019', # ’
    "\x93": u'\u201c', # “
    "\x94": u'\u201d', # ”
    "\x97": u'\u2014'  # —
}

for l in open('file.txt'):
    for c, u in cp1252_to_unicode.items():
        l = l.replace(c, u)

```

But attempting to do the replace this way results in a UnicodeDecodeError being raised, e.g.:

```
"\x85".replace("\x85", u'\u2026')
UnicodeDecodeError: 'ascii' codec can't decode byte 0x85 in position 0: ordinal not in range(128)

```

Any ideas for how to deal with this?

Answer: 
With thanks to jsbueno and a whack of other Google searches and other pounding I solved it this way.

```
#The following works very well but it does not allow for any attempts to FIX the data.
xmlText = unicode(xmlText, errors='replace').replace(u"\uFFFD", "?")

```

This version allows for a limited opportunity to repair invalid characters. Unknown characters are replaced with a safe value.

```
import codecs    
replacement = {
   '85' : '...',           # u'\u2026' ... character.
   '96' : '-',             # u'\u2013' en-dash
   '97' : '-',             # u'\u2014' em-dash
   '91' : "'",             # u'\u2018' left single quote
   '92' : "'",             # u'\u2019' right single quote
   '93' : '"',             # u'\u201C' left double quote
   '94' : '"',             # u'\u201D' right double quote
   '95' : "*"              # u'\u2022' bullet
}

#This is is more complex but allows for the data to be fixed.
def mixed_decoder(unicodeError):
    errStr = unicodeError[1]
    errLen = unicodeError.end - unicodeError.start
    nextPosition = unicodeError.start + errLen
    errHex = errStr[unicodeError.start:unicodeError.end].encode('hex')
    if errHex in replacement:
        return u'%s' % replacement[errHex], nextPosition
    return u'%s' % errHex, nextPosition   # Comment this line out to get a question mark
    return u'?', nextPosition

codecs.register_error("mixed", mixed_decoder)

xmlText = xmlText.decode("utf-8", "mixed")

```

Basically I attempt to turn it into utf8. For any characters that fail I just convert it to HEX so I can display or look it up in a table of my own.

This is not pretty but it does allow me to make sense of messed up data
Just came into this today, so here is my problem and my own solution:

```
original_string = 'Notifica\xe7\xe3o de Emiss\xe3o de Nota Fiscal Eletr\xf4nica.'

def mixed_decoding(s):
    output = ''
    ii = 0
    for c in s:
        if ii <= len(s)-1:
            if s[ii] == '\\' and s[ii+1] == 'x':
                b = s[ii:ii+4].encode('ascii').decode('unicode-escape')
                output = output+b
                ii += 3
            else:
                output = output+s[ii]
        ii += 1
    print(output)
    return output

decoded_string = mixed_decoding(original_string)

```

Now it prints:

>>> Notificação de Emissão de Nota Fiscal Eletrônica.