CLI tools hidden in the Python standard library
Speaking of hidden Python tools, I&#x27;m a big fan of re.Scanner[0]. It&#x27;s a regex-based tokenizer[1] in the `re` module, that for reasons is completely missing from any official documentation.<p>You give it a pattern for each token type, and a function to be called on each match, and you get back a list of processed tokens.<p>Importantly, it processes the list in one pass and ensures the matches are contiguous, where a naive `re.findall` with capture groups will ignore unmatched characters. You also get a reference to the running scanner, so you can record the location of the match for reporting errors.<p><pre><code>    import re
    scanner = re.Scanner([
      (r&quot;[0-9]+&quot;,       lambda scanner, token:(&quot;INTEGER&quot;, int(token))),
      (r&quot;[a-z_]+&quot;,      lambda scanner, token:(&quot;IDENTIFIER&quot;, token)),
      (r&quot;[,.]+&quot;,        lambda scanner, token:(&quot;PUNCTUATION&quot;, token)),
      (r&quot;\s+&quot;, None), # None == skip token.
    ])

    results, remainder = scanner.scan(&quot;45 pigeons, 23 cows, 11 spiders.&quot;)
    assert not remainder
    print(results)

    [(&#x27;INTEGER&#x27;, 45),
     (&#x27;IDENTIFIER&#x27;, &#x27;pigeons&#x27;),
     (&#x27;PUNCTUATION&#x27;, &#x27;,&#x27;),
     (&#x27;INTEGER&#x27;, 23),
     (&#x27;IDENTIFIER&#x27;, &#x27;cows&#x27;),
     (&#x27;PUNCTUATION&#x27;, &#x27;,&#x27;),
     (&#x27;INTEGER&#x27;, 11),
     (&#x27;IDENTIFIER&#x27;, &#x27;spiders&#x27;),
     (&#x27;PUNCTUATION&#x27;, &#x27;.&#x27;)]
</code></pre>
[0]: <a href="https:&#x2F;&#x2F;stackoverflow.com&#x2F;a&#x2F;693818&#x2F;252218" rel="nofollow noreferrer">https:&#x2F;&#x2F;stackoverflow.com&#x2F;a&#x2F;693818&#x2F;252218</a><p>[1]: <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Lexical_analysis#Tokenization" rel="nofollow noreferrer">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Lexical_analysis#Tokenization</a>