jeremyalummitedu jeremy hylton tp tim peters writes false positive rate 03 finally scrub bunch previously unnoticed spam inbox collections 1100 messages tp mean trained 1100 total collections 1100 messages trained 11005 messages tp cant guess youre good position start adding tp headers analysis though example easy start tp would uncomment headercounting lines tokenize tp look anthony likely valuable thing tp missing special parsing tagging received tp headers tried anthony stuff didnt make appreciable difference could see staring false negative rate numbers big enough quick eyeball suffices tried dirt simple tokenizer headers tokenize words header emitted like hdr word worked well received date headers helped classifier discover spam old ham new tried slightly complex one skipped received data xfrom_ contained timestamps also skipped xvm headers mail reader added class mytokenizertokenizer skip received 1 date 1 xfrom_ 1 def tokenize_headersself msg k v msgitems k klower k selfskip kstartswithxvm continue w subject_word_refindallv tokenize_wordw yield ss k moderately better false negative rate 721 tests performed far versus 1128 previous test run used timtest header tokenizer interesting see best descriminators ham discriminators theres single spamindicator list discriminators header fields one thing note presence mailmangenerated headers strong nonspam indicator matches intuition got awful lot mailmangenerated mail lists pretty good surpressing spam thing get lot ham people use xemacs thats probably barry guido fred one final note looks like many false positives people ive never met questions shakespeare often start stuff like dear sirmadam may please take precious time ask help find solution problem worrying greatly old science student guess reads lot like spam jeremy 238 hams 221 spams false positive 210084033613 false negative 950226244344 new false positives new false negatives best discriminators xmailscannerclean 671 00483425 xspamstatusin_rep_to 679 001 deliveredtoskips 10 691 00829876 xmailerlucid 699 001 xmailerxemacs 699 001 xmailerpatch 699 001 xmailerunder 709 001 xmailscannerfound 716 00479124 cczopecom 718 001 ill 750 001 referencesskip1 20 767 001 rossum 795 001 xspamstatusskips 10 825 001 van 850 001 http0zope 869 001 email addrzope 883 001 frompythonorg 895 001 tojeremy 902 0185401 zope 984 001 listarchiveskipm 10 1058 001 listsubscribeskipm 10 1058 001 listunsubscribeskipm 10 1058 001 fromzopecom 1098 001 returnpathzopecom 1115 001 wrote 1129 001 jeremy 1150 001 email addrpython 1257 001 xmailmanversion2013 1311 001 xmailmanversion101270 1395 001 python 1401 001 fri 6 sep 2002 200014 0400 spambayes understanding high false negative rate