Question: I've written a small and simple tokenizer but without use of regular expressions. 

I starts at first index and iterates through every character until end and creates the required tokens. 

I showed it to a colleague that said it would've been much simpler to do /that/ with regex without going into any depths. 

So should I rewrite and expect it to be "better"?

Answer: 
Usually the tokenization can be done with a finite state machine, which is equivalent to canonical regular expressions. If you make sane regexps it will be much easier to read and maintain than a homebrewed FSA. Use tools like flex or jflex and such, they will compile the regexps into minimal FSAs, giving very good performance. Doing this manually should only be done as an exercise. 

Lexers exists in several implementations, quite possibly for your favourite language.
I don't think so. Regex must be very feature-rich, and because of it your program may run slower.