Question: I've written a small and simple tokenizer but without use of regular expressions. 

I starts at first index and iterates through every character until end and creates the required tokens. 

I showed it to a colleague that said it would've been much simpler to do /that/ with regex without going into any depths. 

So should I rewrite and expect it to be "better"?

Answer: 
I don't think so. Regex must be very feature-rich, and because of it your program may run slower.
Depends very much on the language parsed and your definition of "better".