text_combined: timonecomcastnet tim peters jeremy hylton think one step towards deployment creating reusable tokenizer mail messages current codebase doesnt expose easytouse easytocustomize tokenizer tokenize couldnt easier use takes string argument produces stream tokens whether via explicit list generator tuple doesnt matter tokenize functions gbayespy timtestpy freely interchangeable way note evidence support customizable tokenizer would good would ways customization could helpful thats research issue work done timtest module seems contain enormous body practical knowledge parse mail messages module wasnt designed reuse thats partly failure imagination splitting knowledge tokenization large block cutandpaste done change timtoken import tokenize top use tokenizer want make easier still feel free check something better id like see module take single message collection messages tokenize one msg msgstream classes timtestpy start hard anything truly useful people use sorts different physical representations email msgs mboxes various formats one file per folder one file per msg skips gzipped gimmick youre python coder find easy change guts msg msgstream handle peculiar scheme defining interfaces guys done id like see tokenize customizable tim exclude headers test data particular biases test data people test data without biases ought able customize tokenizer include exclude others sounds like bottomless pit theres easier way customize edit code readmetxt still says though massive refactoring would help hop fri 06 sep 2002 133555 0400 spambayes deployment