title: base85 overflow error during decoding of base85 encoded string
body: <p>I need to embed binary data into XML files, so I've chosen to use base85 encoding for this.</p>
  
  <p>I have a large bytearray that's filled with the output of calls to <code>struct.pack()</code> via <code>bytearray.extend(struct.pack(varying_data))</code>. It then gets compressed with <code>zlib</code> and encoded with <code>base64.b85encode()</code>.</p>
  
  <p>This worked all the time, but on a single input file, there is the following strange error:</p>
  
  <pre class="lang-py prettyprint-override"><code>ValueError: base85 overflow in hunk starting at byte 582200`
  </code></pre>
  
  <p>I then modified base64.py to print out which value the current chunk has and which bytes it consists of. The input chunk is <code>b'||a|3'</code> and its value is <strong>4.331.076.573</strong>, which is bigger than <strong>256^4 = 4.294.967.296</strong> and thus can't be represented by four bytes (that's where the error comes from).</p>
  
  <p><strong>But the thing I don't understand is: how can this happen?</strong></p>
  
  <p>This is the important part of the code:</p>
  
  <pre class="lang-py prettyprint-override"><code>elif isinstance(self.content, (bytes, bytearray)):
      base85 = zlib.compress(self.content, 9)
  
      # pad=False doesn't make a difference here
      base85 = base64.b85encode(base85, pad=True).decode()
  
      base85 = escape_xml(base85)
  
      file.write(base85)
  </code></pre>
  
  <pre class="lang-py prettyprint-override"><code>def escape_xml(text):
  
      text = text.replace("&amp;", "&amp;amp;")
      text = text.replace("&lt;", "&amp;lt;")
      text = text.replace("&gt;", "&amp;gt;")
      text = text.replace("\"", "&amp;quot;")
      text = text.replace("'", "&amp;apos;")
  
      return text
  
  </code></pre>
  
  <p>And the code for decoding:</p>
  
  <pre class="lang-py prettyprint-override"><code>def decode_binary_data(data):
      data = unescape_xml(data)
  
      # Remove newline for mixed content support (does not apply in this case)
      data = data.split("\n", 1)[0]
  
      # Error!
      data = base64.b85decode(data)
  
      return zlib.decompress(data)
  
  def unescape_xml(text):
      text = text.replace("&amp;quot;", "\"")
      text = text.replace("&amp;apos;", "'")
      text = text.replace("&amp;lt;", "&lt;")
      text = text.replace("&amp;gt;", "&gt;")
      text = text.replace("&amp;amp;", "&amp;")
  
      return text
  </code></pre>
  
  <p>Base85 can theoretically work with <strong>85^5 = 4.437.053.125</strong> possible combinations, but as it gets input from bytes I'm wondering how this is even possible. Does this come from the compression? That shouldn't be the problem as encoding and decoding should be symmetrical. If it is the problem, how to compress the data anyway?</p>
  
  <p>Choosing Ascii85 instead (<code>a84encode()</code>) works, but I think that this doesn't really solve the problem, maybe it fails in other cases?</p>
  
  <p>Thank you for your help!</p>