timonecomcastnet tim peters guido meant say 099 clues cancelled 001 clues thats wrong looks havent grokked part code yet one way 16 clues seems classifier basically ended counting way 099 001 clues others made list thought looking clues values apparently found none werent exactly 05 theres brief discussion definition max_discriminators clues prob min_spamprob max_spamprob saved min max lists clues fed nbest heap shorter min max lists cancels number clues longer list whatever remains longer list anything fed nbest heap max_discriminators case max_discriminators clues enter final probability calculation min max lists go list clues else youd clue massive cancellation occurring massive cancellation may yet turn hook signal manual review needed specific case excess clues longer max_spamprob list pushed everything else nbest heap thats didnt see anything 001 099 adding special lists outcome faced many 001 099 clues often coin toss whichever flavor happened appear max_discriminators2 1 times first determined final outcome sure sets record longest list cancelling extreme clues happened longest one quite similar ones beat tokenization scheme folds case ignores punctuation strips trailing words saves word bigrams word unigrams turned lowprobability long spam list 410 001 clues 125 099 clues yikes wonder theres anything learn looking clues html heavily markedup html ads sidebar body text serious discussion oo soft coding lots highly technical words clues including zope zeo matter often says zope gets one 001 clue ditto zeo contrast html markup many unique words get 099 btw clear case assumption conditionallyindependent word probabilities utterly bogus eg probability appears message highly correlated prob appearing treating independent naive bayes grossly misjudges probability appear thing get return something actually computed read html section tokenizerpy start ive investigating would work best mailing lists hosted pythonorg html decorations far strong clue justify ignoring specific context havent done anything geared toward personal email including case nonmailinglist email happens go pythonorg id prefer strip html tags everything last time tried still bad effects error rates corpora full test results without html tag stripping included html comment block comment block also says xxx another way found slash fn rate decision xxx strip html htmlonly msgs revisited weve since done several things gave significant fn rate reductions test minablebutunmined header lines corpus left almost apart mime decorations appear headers bodies like contenttype header lines base tokenizer looks subject xmailer organization start different corpus make progress would need different data yes ham polluted mailman header decorations may may able clean fudging data mortal sin havent changed byte far spam polluted header clues fellow collected particular skip received headers suspect theyre going valuable real life example dont even catch undisclosed recipients header sorry following structure multipartmixed textplain brief text plus urls texthtml long html copied website ah explains html tags didnt get stripped id offer add optional argument tokenize theyd get stripped gets glossed third time would feel much like loss seems confused jeremy didnt use trained classifier pickle trained classifier scratch corpora think still corpus size reported tests ran random samples 220 spams 220 hams corpus means training sets sizes well predicting sets sizes harm error rates error rates saw still much better jeremy reported using 500 ah full test run finished tokenization scheme folds case ignores punctuation strips trailing words saves word bigrams word unigrams code tokenize everything body lastw w word_refindalltext n lenw make sure range matches tokenize_word 3 n 12 w1 w w1 yield w lastw yield lastw w lastw w elif n 3 lastw tokenize_wordw yield word_re recompilerwx80xff least doubled process size whats done helped fn rate significantly probably hurt fp rate fp rate low 4000 hams per run confident changes small absolute magnitude 0025 single message fp table false positive percentages 0000 0000 tied 0000 0075 lost 0 0050 0125 lost 15000 0025 0000 10000 0075 0025 6667 0000 0050 lost 0 0100 0175 lost 7500 0050 0050 tied 0025 0050 lost 10000 0025 0000 10000 0050 0125 lost 15000 0050 0025 5000 0050 0050 tied 0000 0025 lost 0 0000 0025 lost 0 0075 0050 3333 0025 0050 lost 10000 0000 0000 tied 0025 0100 lost 30000 0050 0150 lost 20000 5 times tied 4 times lost 11 times total unique fp went 13 21 false negative percentages 0327 0218 3333 0400 0218 4550 0327 0218 3333 0691 0691 tied 0545 0327 4000 0291 0218 2509 0218 0291 lost 3349 0654 0473 2768 0364 0327 1016 0291 0182 3746 0327 0254 2232 0691 0509 2634 0582 0473 1873 0291 0255 1237 0364 0218 4011 0436 0327 2500 0436 0473 lost 849 0218 0218 tied 0291 0255 1237 0254 0364 lost 4331 15 times tied 2 times lost 3 times total unique fn went 106 94 sun 08 sep 2002 031849 0400 spambayes test sets