Show HN: Faster UTF-8 validator
Good stuff.<p>The key to a solid claim, especially for something as bold as &quot;fastest in the world&quot;, is a complete specification of the inputs to the benchmark.<p>You mention random ASCII bytes and &quot;random UTF-8 bytes&quot;. The former is definitely an really important case but also the least interesting. I can write on a napkin a UTF-8-but-is-actually-ASCII decoder that approaches 256 bytes a cycle cached (with a fallback routine when the ASCII assumption fails).<p>So then what about the random UTF-8 bytes. What does it mean? Do you generate random bytes and then exclude invalid sequences? Do you generate a uniform random codepoint in the 21-bit codepoint space and then concert it to utf-8?<p>At a minimum it would good to see a benchmark with random distributions that approximate common languages.<p>The true fastest decoders on such realistic data will be adaptive and more complicated than yours.