tokenize is a part of python distribution, and intended to parse python source code. Is this actually a good tool for your problem? Have you tried nltk?
with tokenize I know you can split text into individual words, but I am confused on how to add characters to indicate the beginning and end of sentences after tokenizing. In my case I want to put ^ to indicate the beginning of the sentence and $ to indicate the end of the sentence. I am asking because I am trying to implement bigram probability models and this is for a school assignment, which is why this is a reinvent the wheel problem.
Is there a Python function to mark the beginning and end of sentences with a specific character after tokenizing?