Q: Determining E-mail character sets My boss assigned me a task of displaying all international characters correctly on our e-mail server that runs on our frontend website. The website uses asp-classic.
I looked at our mail_body.asp code to see how the server was handling the text files received from POP3 and this is what I found:
Set bobj = Server.CreateObject("Basp21")
If InStr(UCase(filecontent),"?UTF-8?") > 0 or InStr(UCase(filecontent),"CHARSET=""UTF-8""" ) > 0 Then
    bobj.CodePage = 65001
ElseIf  InStr(UCase(filecontent),"EUC-KR") or InStr(UCase(filecontent),"KS_C_5601-1987") > 0 or InStr(UCase(filecontent),"CONTENT-TRANSFER-ENCODING: BASE64") > 0 Then
    bobj.CodePage = 949
    'response.Write "euc-kr"
    'response.Write "UTF-8"
ElseIf InStr(UCase(filecontent),"CHARSET=JOHAB") > 0 Then
    bobj.CodePage = 1361
    'response.Write "JOHAB"
ElseIf InStr(UCase(filecontent),"CHARSET=X-MAC-KOREAN") > 0 Then
    bobj.CodePage = 10003   
    'response.Write "X-MAC-KOREAN"
ElseIf InStr(UCase(filecontent),"CHARSET=X-EBCDIC-KOREANEXTENDED") > 0 Then
    bobj.CodePage = 20833   
    'response.Write "X-EBCDIC-KOREAN"
ElseIf InStr(UCase(filecontent),"CHARSET=ISO-2022-KR") > 0 Then
    bobj.CodePage = 50225
    'response.Write "ISO-2022-KR"   
ElseIf InStr(UCase(filecontent),"CHARSET=""GB2312""") > 0 or InStr(UCase(filecontent),"CHARSET=GB2312") Then
    bobj.CodePage = 936
Else
    bobj.CodePage = 65001
End If

...
outarray=bobj.ReadMail(contents,"subject:to:from:date:X-OriginalArrivalTime:",dirname)

I think it's safe to say that above code is not elegant at all, but it gets the job done for the most part. It's a Korean website, so the code is looking for headers that are non-unicode but is written in Korean. Otherwise, it assumes that the e-mail received is UTF-8.
So whenever there are mails with character set other than UTF-8 and any of the defined Korean code pages in the code, the parsed text gets all jumbled up.
Because my boss have asked me to "make the e-mail content display all international characters", the only solution I can think of is to exhaustively check for all character sets listed here which the header can contain and set the code page appropriately but this doesn't seem to be a good solution.
I'm wondering what the practical approach would be here to take. Many email services like hotmail and gmail have no problem displaying international characters so I'd like to know how they approached this kind of problem, if possible.

A: The practical approach is to parse out the value of charset=* and convert the content to some unified internal encoding. Pseudocode (since I'm not an ASP person at all):
charset  = parseMailHeaders(mail.headers, 'charset')  // e.g. 'ISO-2022-KR'
mailBody = convertEncoding(charset, 'UTF-16', mail.body)

The hypothetical parseMailHeaders extracts the value of charset=* from the appropriate mail header. Depending on what values your hypothetical convertEncoding function accepts you may need to do some normalization on the extracted charset value, but that's basically it. From then on you know the content is UTF-16 encoded and can treat it as such.
