title: What characters must be escaped in an HTTP query string?
body: <p>This question concerns the characters in the query string portion of the URL, which appear after the <code>?</code> mark character.</p>
  <p>Per <a href="http://en.wikipedia.org/wiki/Query_string" rel="noreferrer">Wikipedia</a>, certain characters are left as is and others are encoded (usually with a <code>%</code> escape sequence).</p>
  <p>I've been trying to track this down to actual specifications, so that I understand the justification behind every bullet point in that Wikipedia page.</p>
  <p><strong>Contradiction Example 1:</strong></p>
  <p>The <a href="http://www.w3.org/TR/REC-html40/interact/forms.html#form-content-type" rel="noreferrer">HTML specification</a> says to encode space as <code>+</code> and defers the rest to <a href="http://www.ietf.org/rfc/rfc1738.txt" rel="noreferrer">RFC1738</a>. However, this RFC says that <code>~</code> is unsafe and furthermore that &quot;[a]ll unsafe characters must always be encoded within the URL&quot;. This seems to contradict Wikipedia.</p>
  <p>In practice, IE8 encodes <code>~</code> in the query strings it generates, while FF3 leaves it as is.</p>
  <p><strong>Contradiction Example 2:</strong></p>
  <p>Wikipedia states that all characters that it does not mention must be encoded. <code>!</code> is not mentioned in Wikipedia. But <a href="http://www.ietf.org/rfc/rfc1738.txt" rel="noreferrer">RFC1738</a> states that <code>!</code> is a &quot;special&quot; character and &quot;may be used unencoded&quot;. This seems to contradict Wikipedia which says that it must be encoded.</p>
  <p>In practice, IE8 encodes <code>!</code> in the query strings it generates, while FF3 leaves it as is.</p>
  <p><em>I understand that the moral of this is probably going to be to encode those characters that are in doubt between Wikipedia and the specifications. Perhaps even going as far as encoding everything that is not [A-Za-z0-9]. I would just like to know the actual standards on this.</em></p>
  <p><strong>Conclusions</strong></p>
  <p>The algorithm described on Wikipedia encodes precisely those characters which are not <a href="https://www.rfc-editor.org/rfc/rfc3986#section-2.3" rel="noreferrer">RFC3986 unreserved characters</a>. That is, it encodes all characters other than alphanumerics and <code>-._~</code>. As a special case, space is encoded as <code>+</code> instead of <code>%20</code> per RFC3986.</p>
  <p>Some applications use an older RFC. For comparison, the <a href="https://www.rfc-editor.org/rfc/rfc2396#page-9" rel="noreferrer">RFC2396 unreserved characters</a> are alphanumerics and <code>!'()*-._~</code>.</p>
  <p>For comparison, the <a href="http://dev.w3.org/html5/spec/Overview.html#application-x-www-form-urlencoded-encoding-algorithm" rel="noreferrer">HTML5 working draft algorithm</a> encodes all characters other than alphanumerics and <code>*-._</code>. The special case encoding for space remains <code>+</code>. Notable differences are that <code>*</code> is not encoded and <code>~</code> is encoded. (Technically, this handling of <code>*</code> is compatible with RFC3986 even though <code>*</code> is in <code>reserved</code> because it is in the <code>sub-delims</code> which are allowed in the <code>query</code> production.)</p>