Show HN: Big List of Naughty Strings for testing user-input data
Most of what I do involves the messy world of text, and I think this is a great resource. I wish the software I depended on tested against it.I can think of a few more cases that I've seen cause havoc:- U+FEFF in the middle of a string (people are used to seeing it at the beginning of a string, because Microsoft, but elsewhere it may be more surprising)- U+0 (it's encoded as the null byte!)- U+1B (the codepoint for "escape")- U+85 (Python's "codecs" module thinks this is a newline, while the "io" module and the Python 3 standard library don't)- U+2028 and U+2029 (even weirder linebreaks that cause disagreement when used in JSON literals)- A glyph with a million combining marks on it, but not in NFC order (do your Unicode algorithms use insertion sort?)- The sequence U+100000 U+010000 (triggers a weird bug in Python 3.2 only)- "Forbidden" strings that are still encodable, such as U+FFFF, U+1FFFF, and for some reason U+FDD0People should also test what happens with isolated surrogate codepoints, such as U+D800. But these can't properly be encoded in UTF-8, so I guess don't put them in the BLNS. (If you put the fake UTF-8 for them in a file, the best thing for a program to do would be to give up on reading the file.)