Question: I have a file which is mostly UTF-8, but some Windows-1252 characters have also found their way in.

I created a table to map from the Windows-1252 (cp1252) characters to their Unicode counterparts, and would like to use it to fix the mis-encoded characters, e.g.

```
cp1252_to_unicode = {
    "\x85": u'\u2026', # …
    "\x91": u'\u2018', # ‘
    "\x92": u'\u2019', # ’
    "\x93": u'\u201c', # “
    "\x94": u'\u201d', # ”
    "\x97": u'\u2014'  # —
}

for l in open('file.txt'):
    for c, u in cp1252_to_unicode.items():
        l = l.replace(c, u)

```

But attempting to do the replace this way results in a UnicodeDecodeError being raised, e.g.:

```
"\x85".replace("\x85", u'\u2026')
UnicodeDecodeError: 'ascii' codec can't decode byte 0x85 in position 0: ordinal not in range(128)

```

Any ideas for how to deal with this?

Answer: 
With thanks to jsbueno and a whack of other Google searches and other pounding I solved it this way.

```
#The following works very well but it does not allow for any attempts to FIX the data.
xmlText = unicode(xmlText, errors='replace').replace(u"\uFFFD", "?")

```

This version allows for a limited opportunity to repair invalid characters. Unknown characters are replaced with a safe value.

```
import codecs    
replacement = {
   '85' : '...',           # u'\u2026' ... character.
   '96' : '-',             # u'\u2013' en-dash
   '97' : '-',             # u'\u2014' em-dash
   '91' : "'",             # u'\u2018' left single quote
   '92' : "'",             # u'\u2019' right single quote
   '93' : '"',             # u'\u201C' left double quote
   '94' : '"',             # u'\u201D' right double quote
   '95' : "*"              # u'\u2022' bullet
}

#This is is more complex but allows for the data to be fixed.
def mixed_decoder(unicodeError):
    errStr = unicodeError[1]
    errLen = unicodeError.end - unicodeError.start
    nextPosition = unicodeError.start + errLen
    errHex = errStr[unicodeError.start:unicodeError.end].encode('hex')
    if errHex in replacement:
        return u'%s' % replacement[errHex], nextPosition
    return u'%s' % errHex, nextPosition   # Comment this line out to get a question mark
    return u'?', nextPosition

codecs.register_error("mixed", mixed_decoder)

xmlText = xmlText.decode("utf-8", "mixed")

```

Basically I attempt to turn it into utf8. For any characters that fail I just convert it to HEX so I can display or look it up in a table of my own.

This is not pretty but it does allow me to make sense of messed up data
Good solution that of @jsbueno, but there is no need of global variable `last_position`, see:

```py
def mixed_decoder(error: UnicodeError) -> (str, int):
     bs: bytes = error.object[error.start: error.end]
     return bs.decode("cp1252"), error.start + 1

import codecs
codecs.register_error("mixed", mixed_decoder)

a = "maçã".encode("utf-8") + "maçã".encode("cp1252")
# a = b"ma\xc3\xa7\xc3\xa3ma\xe7\xe3"

s = a.decode("utf-8", "mixed")
# s = "maçãmaçã"

```