text_combined: timonecomcastnet tim peters tp im reading trained 220 spam tp 220 ham thats less 10 sizes tp training sets ive using please try experiment train tp 550 test 550 jeremy helps lot possibly checked change classifierpy overnight getting rid mincount gave major improvement fn rate independent tokenization results stock tokenizer unsure stock tokenizer means example might mean tokenizertokenize mboxtestmytokenizertokenize training homejeremymailspam 8 644 hams 557 spams 0000 10413 1398 6104 1398 5027 training homejeremymailspam 0 644 hams 557 spams 0000 8259 1242 2873 1242 5745 training homejeremymailspam 3 644 hams 557 spams 1398 5206 1398 4488 0000 9336 training homejeremymailspam 0 644 hams 557 spams 1553 5206 1553 5027 0000 9874 total false pos 139 539596273292 total false neg 970 435368043088 note rates remain much higher got using 220 ham 220 spam remains mystery results tokenizer looks headers except date received xfrom_ unsure means example looks might mean enabled anthonys countthem gimmick andor youre tokenizing andor training homejeremymailspam 8 644 hams 557 spams 0000 7540 0932 4847 0932 3232 training homejeremymailspam 0 644 hams 557 spams 0000 7181 0621 2873 0621 4847 training homejeremymailspam 3 644 hams 557 spams 1087 4129 1087 3052 0000 6822 training homejeremymailspam 0 644 hams 557 spams 0776 3411 0776 3411 0000 6463 total false pos 97 376552795031 total false neg 738 331238779174 safe conclude avoiding cleverness headers good thing since dont know exactly cant guess seemed show something clever headers helped numbers better numbers right assuming override whats tokenizertokenizertokenize_headers routine didnt call base tokenizertokenize_headers youre missing carefully tested treatment header fields adding many dozens header fields theres question adding header fields help tokenizertokenizertokenize_headers doesnt testing corpora cant add headers without getting major benefits bogus reasons apart said youre skipping received several accounts may valuable header fields im meaning tokenizertokenizertokenize_headers skipping reason explained offline week two ago neil schemenauer reported good results scheme ip_re recompilerd13d13d13d13 header msgget_allreceived ip ip_refindallheader parts ipsplit n range1 5 yield received joinpartsn makes lot sense checked left disabled sat 07 sep 2002 154518 0400 spambayes understanding high false negative rate