title: Pythonic way to implement a tokenizer
body: <p>I'm going to implement a tokenizer in Python and I was wondering if you could offer some style advice?</p>
  
  <p>I've implemented a tokenizer before in C and in Java so I'm fine with the theory, I'd just like to ensure I'm following pythonic styles and best practices.</p>
  
  <p>Listing Token Types:</p>
  
  <p>In Java, for example, I would have a list of fields like so:</p>
  
  <pre><code>public static final int TOKEN_INTEGER = 0
  </code></pre>
  
  <p>But, obviously, there's no way (I think) to declare a constant variable in Python, so I could just replace this with normal variable declarations but that doesn't strike me as a great solution since the declarations could be altered.</p>
  
  <p>Returning Tokens From The Tokenizer:</p>
  
  <p>Is there a better alternative to just simply returning a list of tuples e.g. </p>
  
  <pre><code>[ (TOKEN_INTEGER, 17), (TOKEN_STRING, "Sixteen")]?
  </code></pre>
  
  <p>Cheers,</p>
  
  <p>Pete</p>