Resources for lexing, tokenising and parsing in python
<p>Can people point me to resources on lexing, parsing and tokenising with Python?</p>

<p>I'm doing a little hacking on an open source project (<a href="http://www.hotwire-shell.org/">hotwire</a>) and wanted to do a few changes to the code that <a href="http://code.google.com/p/hotwire-shell/source/browse/trunk/hotwire/externals/shlex.py">lexes</a>, <a href="http://code.google.com/p/hotwire-shell/source/browse/trunk/hotwire/command.py">parses and tokenises</a> the commands entered into it.  As it is real working code it is fairly complex and a bit hard to work out.</p>

<p>I haven't worked on code to lex/parse/tokenise before, so I was thinking one approach would be to work through a tutorial or two on this aspect.  I would hope to learn enough to navigate around the code I actually want to alter.  Is there anything suitable out there?  (Ideally it could be done in an afternoon without having to buy and read the dragon book first ...)</p>

<p><strong>Edit:</strong> (7 Oct 2008) None of the below answers quite give what I want.  With them I could generate parsers from scratch, but I want to learn how to write my own basic parser from scratch, not using lex and yacc or similar tools.  Having done that I can then understand the existing code better.  </p>

<p>So could someone point me to a tutorial where I can build a basic parser from scratch, using just python?</p>

<p>I'm a happy user of <a href="http://www.dabeaz.com/ply/">PLY</a>. It is a pure-Python implementation of Lex &amp; Yacc, with lots of small niceties that make it quite Pythonic and easy to use. Since Lex &amp; Yacc are the most popular lexing &amp; parsing tools and are used for the most projects, PLY has the advantage of standing on giants' shoulders. A lot of knowledge exists online on Lex &amp; Yacc, and you can freely apply it to PLY.</p>

<p>PLY also has a good <a href="http://www.dabeaz.com/ply/ply.html">documentation page</a> with some simple examples to get you started. </p>

<p>For a listing of lots of Python parsing tools, see <a href="http://nedbatchelder.com/text/python-parsers.html">this</a>.</p>
