Unicode normalization for filenames and copied text from pdf:s I have two issues that seem related. When copying text or filenames containing umlauts or e.g. å ä ö, OS X can't seem to handle the characters in a sane way. Apparently this is a well-known problem. For example:


http://support.wandisco.com/index.php?/Knowledgebase/Article/View/401/55/problems-with-umlauts--in-filenames-on-mac-os-x 
https://tex.stackexchange.com/questions/94418/os-x-umlauts-in-utf8-nfd-yield-package-inputenc-error-unicode-char-u8-not


Here is an example of the usual result when copying text from a filename or from inside a pdf in Preview and pasting it in an editor. The first line is the result, the other one is corrected.



The difference is clearly visible, since the current font (Courier Prime) doesn't support the first version.

Is there any way to fix this? Alternatively, is there any OS X-service available to "clean up" text or normalize it in the correct way?
HFS+ requires filenames to be in decomposed form (LATIN SMALL LETTER A + COMBINING DIAERESIS) instead of composed form (LATIN SMALL LETTER A WITH DIAERESIS). You can use iconv to convert text to composed form:

$ echo -n ä | xxd -p
c3a4
$ touch ä
$ ls | tr -d '\n' | xxd -p
61cc88
$ ls | tr -d '\n' | iconv -f utf-8-mac -t utf-8 | xxd -p
c3a4




HFS+ does not use NFD (normal form decomposed). From http://developer.apple.com/library/mac/#qa/qa1173/_index.html:


  Important: The terms used in this Q&amp;A, precomposed and decomposed, roughly correspond to Unicode Normal Forms C and D, respectively. However, most volume formats do not follow the exact specification for these normal forms. For example, HFS Plus (Mac OS Extended) uses a variant of Normal Form D in which U+2000 through U+2FFF, U+F900 through U+FAFF, and U+2F800 through U+2FAFF are not decomposed (this avoids problems with round trip conversions from old Mac text encodings).


Something like this might also work:

python -c 'import unicodedata as ud; print ud.normalize("NFC", u"\N{LATIN SMALL LETTER A}\N{COMBINING DIAERESIS}")'