I agree, proper tokenizers will most likely do that properly. I was just making a crude guess why there might be lookup for a whitespace in the first place.
I certainly had similar issues since I had to implement my own tokenizer for better handling social media text where people forget whitespaces, use emojis or emoticons as sentence separators, etc. Granted the more common problem I had was empty stokens instead of tokens with one or more whitespaces, like in:
'1 2 3     '.split(' ')
# ['1', '2', '', '', '', '', '3']