title: Why is python decode replacing more than the invalid bytes from an encoded string?
body: <p>Trying to decode an invalid encoded utf-8 html page gives different results in
  python, firefox and chrome.</p>
  
  <p>The invalid encoded fragment from test page looks like <code>'PREFIX\xe3\xabSUFFIX'</code></p>
  
  <pre><code>&gt;&gt;&gt; fragment = 'PREFIX\xe3\xabSUFFIX'
  &gt;&gt;&gt; fragment.decode('utf-8', 'strict')
  ...
  UnicodeDecodeError: 'utf8' codec can't decode bytes in position 6-8: invalid data
  </code></pre>
  
  <p><strong>UPDATE</strong>: This question concluded in a <a href="http://bugs.python.org/issue8271" rel="noreferrer" title="str.decode(&#39;utf8&#39;, &#39;replace&#39;) -- conformance with Unicode 5.2.0">bug report</a> to Python unicode component. The Issue is reported to be fixed in Python 2.7.11 and 3.5.2.</p>
  
  <hr>
  
  <p>What follows is the replacement policies used to handle decoding errors in
  Python, Firefox and Chrome.  Note how they differs, and specially how
  python builtin removes the valid <code>S</code> (plus the invalid sequence of bytes).</p>
  
  <h3>Python</h3>
  
  <p>The builtin <code>replace</code> error handler replaces the invalid <code>\xe3\xab</code> plus the
  <code>S</code> from <code>SUFFIX</code> by U+FFFD</p>
  
  <pre><code>&gt;&gt;&gt; fragment.decode('utf-8', 'replace')
  u'PREFIX\ufffdUFFIX'
  &gt;&gt;&gt; print _
  PREFIX�UFFIX
  </code></pre>
  
  <h3>Browsers</h3>
  
  <p>To tests how browsers decode the invalid sequence of bytes will use a cgi script:</p>
  
  <pre><code>#!/usr/bin/env python
  print """\
  Content-Type: text/plain; charset=utf-8
  
  PREFIX\xe3\xabSUFFIX"""
  </code></pre>
  
  <p>Firefox and Chrome browsers rendered:</p>
  
  <pre><code>PREFIX�SUFFIX
  </code></pre>
  
  <hr>
  
  <h3>Why builtin <code>replace</code> error handler for <code>str.decode</code> is removing the <code>S</code> from <code>SUFFIX</code></h3>
  
  <p><em>(Was UPDATE 1)</em></p>
  
  <p>According to wikipedia <a href="http://en.wikipedia.org/wiki/Utf8" rel="noreferrer">UTF-8</a> (thanks mjv),
  the following ranges of bytes are used to indicate the start of a sequence of
  bytes</p>
  
  <ul>
  <li>0xC2-0xDF : Start of 2-byte sequence</li>
  <li>0xE0-0xEF : Start of 3-byte sequence</li>
  <li>0xF0-0xF4 : Start of 4-byte sequence</li>
  </ul>
  
  <p><code>'PREFIX\xe3\abSUFFIX'</code> test fragment has <strong>0xE3</strong>, it instructs python decoder
  that a 3-byte sequence follows, the sequence is found invalid and python
  decoder ignores the whole sequence including <code>'\xabS'</code>, and continues after it
  ignoring any possible correct sequence starting in the middle.</p>
  
  <p>This means that for an invalid encoded sequence like <code>'\xF0SUFFIX'</code>, it will
  decode <code>u'\ufffdFIX'</code> instead of <code>u'\ufffdSUFFIX'</code>.</p>
  
  <p>Example 1: Introducing DOM parsing bugs </p>
  
  <pre><code>&gt;&gt;&gt; '&lt;div&gt;\xf0&lt;div&gt;Price: $20&lt;/div&gt;...&lt;/div&gt;'.decode('utf-8', 'replace')
  u'&lt;div&gt;\ufffdv&gt;Price: $20&lt;/div&gt;...&lt;/div&gt;'
  &gt;&gt;&gt; print _
  &lt;div&gt;�v&gt;Price: $20&lt;/div&gt;...&lt;/div&gt;
  </code></pre>
  
  <p>Example 2: Security issues (Also see <a href="http://www.unicode.org/reports/tr36/#Ill-Formed_Subsequences" rel="noreferrer" title="Unicode Security Considerations - Ill formed subsequences">Unicode security considerations</a>):</p>
  
  <pre><code>&gt;&gt;&gt; '\xf0&lt;!-- &lt;script&gt;alert("hi!");&lt;/script&gt; --&gt;'.decode('utf-8', 'replace')
  u'\ufffd- &lt;script&gt;alert("hi!");&lt;/script&gt; --&gt;'
  &gt;&gt;&gt; print _
  �- &lt;script&gt;alert("hi!");&lt;/script&gt; --&gt;
  </code></pre>
  
  <p>Example 3: Remove valid information for a scraping application</p>
  
  <pre><code>&gt;&gt;&gt; '\xf0' + u'it\u2019s'.encode('utf-8') # "it’s"
  '\xf0it\xe2\x80\x99s'
  &gt;&gt;&gt; _.decode('utf-8', 'replace')
  u'\ufffd\ufffd\ufffds'
  &gt;&gt;&gt; print _
  ���s
  </code></pre>
  
  <p>Using a cgi script to render this in browsers:</p>
  
  <pre><code>#!/usr/bin/env python
  print """\
  Content-Type: text/plain; charset=utf-8
  
  \xf0it\xe2\x80\x99s"""
  </code></pre>
  
  <p>Rendered:</p>
  
  <pre><code>�it’s
  </code></pre>
  
  <hr>
  
  <h3>Is there any official recommended way for handling decoding replacements?</h3>
  
  <p><em>(Was UPDATE 2)</em></p>
  
  <p>In a <a href="http://www.unicode.org/review/pr-121.html" rel="noreferrer" title="UTC Public Review 121">public review</a>, the Unicode Technical Committee has opted for option 2
  of the following candidates:</p>
  
  <ol>
  <li>Replace the entire ill-formed subsequence by a single U+FFFD.</li>
  <li>Replace each maximal subpart of the ill-formed subsequence by a single U+FFFD.</li>
  <li>Replace each code unit of the ill-formed subsequence by a single U+FFFD.</li>
  </ol>
  
  <p>UTC Resolution was at 2008-08-29, source: <a href="http://www.unicode.org/review/resolved-pri-100.html" rel="noreferrer">http://www.unicode.org/review/resolved-pri-100.html</a></p>
  
  <p><a href="http://www.unicode.org/review/pr-121.html" rel="noreferrer" title="UTC Public Review 121">UTC Public Review 121</a> also includes an invalid bytestream as example
  <code>'\x61\xF1\x80\x80\xE1\x80\xC2\x62'</code>, it shows decoding results for each
  option.</p>
  
  <pre><code>            61      F1      80      80      E1      80      C2      62
        1   U+0061  U+FFFD                                          U+0062
        2   U+0061  U+FFFD                  U+FFFD          U+FFFD  U+0062
        3   U+0061  U+FFFD  U+FFFD  U+FFFD  U+FFFD  U+FFFD  U+FFFD  U+0062
  </code></pre>
  
  <p>In plain Python the three results are:</p>
  
  <ol>
  <li><code>u'a\ufffdb'</code> shows as <code>a�b</code></li>
  <li><code>u'a\ufffd\ufffd\ufffdb'</code> shows as <code>a���b</code></li>
  <li><code>u'a\ufffd\ufffd\ufffd\ufffd\ufffd\ufffdb'</code> shows as <code>a������b</code></li>
  </ol>
  
  <p>And here is what python does for the invalid example bytestream:</p>
  
  <pre><code>&gt;&gt;&gt; '\x61\xF1\x80\x80\xE1\x80\xC2\x62'.decode('utf-8', 'replace')
  u'a\ufffd\ufffd\ufffd'
  &gt;&gt;&gt; print _
  a���
  </code></pre>
  
  <p>Again, using a cgi script to test how browsers render the buggy encoded bytes:</p>
  
  <pre><code>#!/usr/bin/env python
  print """\
  Content-Type: text/plain; charset=utf-8
  
  \x61\xF1\x80\x80\xE1\x80\xC2\x62"""
  </code></pre>
  
  <p>Both, Chrome and Firefox rendered:</p>
  
  <pre><code>a���b
  </code></pre>
  
  <p>Note that browsers rendered result matches option 2 of PR121 recomendation</p>
  
  <p>While <em>option 3</em> looks easily implementable in python, <em>option 2 and 1</em> are a challenge.</p>
  
  <pre><code>&gt;&gt;&gt; replace_option3 = lambda exc: (u'\ufffd', exc.start+1)
  &gt;&gt;&gt; codecs.register_error('replace_option3', replace_option3)
  &gt;&gt;&gt; '\x61\xF1\x80\x80\xE1\x80\xC2\x62'.decode('utf-8', 'replace_option3')
  u'a\ufffd\ufffd\ufffd\ufffd\ufffd\ufffdb'
  &gt;&gt;&gt; print _
  a������b
  </code></pre>