My url isn't your url
It seems nobody really understands how to handle non-ascii URLs. To get it right for Scrapy (i.e. do same as browsers) we checked how several browsers behave; the results was consistent but surprising, if not crazy. Before sending an URL to a server (e.g. when user clicks a link)  browser does the following:<p>1. Url is parsed into components.<p>2. Domain name is encoded using punycode.<p>3. URL path is encoded to UTF-8 and then percent-escaped.<p>4. Query string and fragment are encoded using <i>page encoding</i> and then percent-escaped. For URLs entered to address bar it seems that UTF-8 is used, not page encoding.<p>5. The result is joined back to a single byte string and sent to a server.<p>So this means parts of a single URL are encoded using 3 different encodings, and one needs to know the encoding of a page URL was extracted from to send it correctly (i.e. to send it the way browsers do). I haven't seen this algorithm stated explicitly anywhere, but this is how all browsers work. It doesn't make any sense.