text_combined: dwooqfuuguserssourceforgenet revision 3155 httpspambayessvnsourceforgenetspambayesrev3155viewrev author montanaro date 20070725 064942 0700 wed 25 jul 2007 log message cvs svn modified paths trunkwebsiteapplicationsht trunkwebsitebackgroundht trunkwebsitecontactht trunkwebsitedeveloperht trunkwebsitedocsht trunkwebsitedownloadht trunkwebsiteunixht trunkwebsitewindowsht added paths trunkwebsiteprefschangeloght removed paths trunkwebsitepresfchangeloght modified trunkwebsiteapplicationsht trunkwebsiteapplicationsht 20070724 000432 utc rev 3154 trunkwebsiteapplicationsht 20070725 134942 utc rev 3155 2513 2513 pythons win32com extensions win32all149 later currently activepython suitable see readmetxt abouthtml file spambayes cvs repositorys outlook2000 directory alternatively use cvs get code go cvs page projects sourceforge site see readmetxt abouthtml file spambayes subversion repositorys outlook2000 directory alternatively use subversion get code go subversion page projects sourceforge site sb_filterpy sb_filter command line tool marking mail ham spam readme includes guide integrating mailer unixonly instructions moment additions welcome currently focuses running sb_filter via procmail 4411 4411 availability download source archive alternatively use cvs get code go cvs page projects sourceforge site alternatively use subversion get code go subversion page projects sourceforge site sb_serverpy sb_server provides pop3 proxy sits mail client real pop3 server marks mail ham spam passes see readme mail ham spam passes see readme sb_server also used sb_uploadpy script procmail similar solution requirements 6312 6312 alternatively run source download source archive alternatively use cvs get code go cvs page projects sourceforge site alternatively use subversion get code go subversion page projects sourceforge site sb_imapfilterpy imapfilter connects imap server marks mail ham spam moving appropriate folders arrives see readme see readme requirements 787 787 availability download source archive alternatively use cvs get code go cvs page projects sourceforge site alternatively use subversion get code go subversion page projects sourceforge site sb_mboxtrainpy application allows train incrementally ham spam 947 947 availability download source archive alternatively use cvs get code go cvs page projects sourceforge site alternatively use subversion get code go subversion page projects sourceforge site sb_notesfilterpy application allows filter lotus notes folders rather like 1124 1124 availability download source archive alternatively use cvs get code go cvs page projects sourceforge site alternatively use subversion get code go subversion page projects sourceforge site modified trunkwebsitebackgroundht trunkwebsitebackgroundht 20070724 000432 utc rev 3154 trunkwebsitebackgroundht 20070725 134942 utc rev 3155 25010 25010 discussions moved spambayes mailing list cvs commit messages cvs commit messages tim peters whacked whole lot useful information cvs commit messages project moved obscure corner python cvs tree theres actually two sources cvs commits python cvs tree actually two sources cvs commits older cvs repository via view cvs entire changelog modified trunkwebsitecontactht trunkwebsitecontactht 20070724 000432 utc rev 3154 trunkwebsitecontactht 20070725 134942 utc rev 3155 347 347 list also unmoderated cvs commit messages go list spambayescheckins subversion commit messages go list spambayescheckins shouldnt send email list program running sourceforge automatically creates sends emails list result code checkins modified trunkwebsitedeveloperht trunkwebsitedeveloperht 20070724 000432 utc rev 3154 trunkwebsitedeveloperht 20070725 134942 utc rev 3155 510 59 developer info want get involved running code project works python 22 python 23 python 24 bleeding edge python code available cvs sourceforge work python 21x earlier ever project works python 22 later available subversion sourceforge work python 21x earlier ever likely youre running python 22 221 youll need separately fetch latest email package get 177 167 youll need version 243 later version 30 later recommended spambayes code also available via cvs download page spambayes code also available via via subversion download page want make suggestions modified trunkwebsitedocsht trunkwebsitedocsht 20070724 000432 utc rev 3154 trunkwebsitedocsht 20070725 134942 utc rev 3155 1013 1013 generally help would great see documentation improvements hints tips scripts recipes anything else related spambayes takes fancy added instructions installing spambayes integrating mail system outlook plugin includes file instructions installing spambayes integrating mail system outlook plugin includes file troubleshooting guide accessed via toolbar note online documentaton always latest source version might correspond exactly version using always start documentation came version installed readmedeveltxt information use people planning developing code based spambayes testingtxt file clues practice statistical testing adapted tim readmedeveltxt information use people planning developing code based spambayes testingtxt file clues practice statistical testing adapted tim comments pythondev also vast number clues notes scattered block comments code modified trunkwebsitedownloadht trunkwebsitedownloadht 20070724 000432 utc rev 3154 trunkwebsitedownloadht 20070725 134942 utc rev 3155 387 387 prerequisites either python 222 cvs build python python 222 subversion build python python 22 221 plus latest email package youve downloaded unpacked source archive regular setuppy build setuppy install dance 10610 1068 instructions geared gnupg commandline weenies suggestions welcome openpgp applications cvs access code currently available sourceforges cvs server see details note due capacity problems sourceforge public cvs servers often run 48 hours behind real cvs servers something sf working improving subversion access code currently available sourceforge subversion server copied trunkwebsiteprefschangeloght rev 3151 trunkwebsitepresfchangeloght trunkwebsiteprefschangeloght rev 0 trunkwebsiteprefschangeloght 20070725 134942 utc rev 3155 00 1905 presourceforge changelog changelog lists commits spambayes projects separate project set see also old cvs repository dont forget date probably want looking current cvs 20020906 0227 tim_one gbayespy 116 testerpy 14 classifierpy 112 cleanarch 13 mboxcountpy 16 rebalpy 14 setuppy 12 splitpy 16 splitnpy 13 timtestpy 118 code moved new sourceforge project spambayes 20020905 1537 tim_one classifierpy 111 added note mincount oddities 20020905 1432 tim_one timtestpy 117 added note word length 20020905 1348 tim_one timtestpy 116 tokenize_word oops awfully permissive took email address tightened also avoided 5graming email addresses w highbit characters false positive percentages 0000 0000 tied 0000 0000 tied 0050 0050 tied 0000 0000 tied 0025 0025 tied 0025 0025 tied 0050 0050 tied 0025 0025 tied 0025 0025 tied 0025 0050 lost 0075 0075 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0000 0000 tied 0025 0025 tied 0050 0050 tied 0 times tied 19 times lost 1 times total unique fp went 7 8 false negative percentages 0764 0691 0691 0655 0981 0945 1309 1309 tied 1418 1164 0873 0800 0800 0763 1163 1163 tied 1491 1345 1200 1127 1381 1345 1454 1490 lost 1164 0909 0655 0582 0655 0691 lost 1163 1163 tied 1200 1018 0982 0873 0982 0909 1236 1127 15 times tied 3 times lost 2 times total unique fn went 260 249 note two losses consist 1 msg difference wins bigger well common 260249 11 spams longer sneak run 4 260 spams used sneak thru 20020905 1151 tim_one classifierpy 110 comment test results moving max_discriminators back 15 doesnt really matter leaving alone 20020905 1002 tim_one classifierpy 19 nowrare pure win changing spamprob work harder find evidence competing 001 099 clues appear left column right false positive percentages 0000 0000 tied 0000 0000 tied 0050 0050 tied 0000 0000 tied 0025 0025 tied 0025 0025 tied 0050 0050 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0075 0075 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0075 0025 0025 0025 tied 0025 0025 tied 0000 0000 tied 0025 0025 tied 0050 0050 tied 1 times tied 19 times lost 0 times total unique fp went 9 7 false negative percentages 0909 0764 0800 0691 1091 0981 1381 1309 1491 1418 1055 0873 0945 0800 1236 1163 1564 1491 1200 1200 tied 1454 1381 1599 1454 1236 1164 0800 0655 0836 0655 1236 1163 1236 1200 1055 0982 1127 0982 1381 1236 19 times tied 1 times lost 0 times total unique fn went 284 260 20020904 1121 tim_one timtestpy 115 augmented spam callback display spams low probability 20020904 0953 tim_one testerpy 13 timtestpy 114 added support simple histograms probability distributions ham spam 20020903 1213 tim_one timtestpy 113 reluctant principle change matter stats take stab removing html decorations plain text msgs see comments plain text msgs puts end false positives due text msgs talking html surprisingly also gets rid false negatives surprisingly introduced another small class false positives due dumbass regexp trick used approximate html tag removal removing pieces text nothing html tags eg happened middle uuencoded py file happened leave behind string looked like spam phrase looked like pile long lines didnt generate tokens nonsense outcome either way false positive percentages 0000 0000 tied 0000 0000 tied 0050 0050 tied 0000 0000 tied 0025 0025 tied 0025 0025 tied 0050 0050 tied 0025 0025 tied 0025 0025 tied 0000 0025 lost 0075 0075 tied 0050 0025 0025 0025 tied 0000 0025 lost 0050 0075 lost 0025 0025 tied 0025 0025 tied 0000 0000 tied 0025 0025 tied 0050 0050 tied 1 times tied 16 times lost 3 times total unique fp went 8 9 false negative percentages 0945 0909 0836 0800 1200 1091 1418 1381 1455 1491 lost 1091 1055 1091 0945 1236 1236 tied 1564 1564 tied 1236 1200 1563 1454 1563 1599 lost 1236 1236 tied 0836 0800 0873 0836 1236 1236 tied 1273 1236 1018 1055 lost 1091 1127 lost 1490 1381 12 times tied 4 times lost 4 times total unique fn went 292 284 20020903 0657 tim_one classifierpy 18 added new xspamprob method computes combined probability correctly long comment block explaining happened tried theres something worth pursuing greatly improves false negative rate change alone pushes many marginal hams spam camp 20020903 0523 tim_one timtestpy 112 made skip tokens shorter added surprising treatment organization headers tiny fn benefit tiny cost change fp stats false negative percentages 1091 0945 0945 0836 1236 1200 1454 1418 1491 1455 1091 1091 tied 1127 1091 1236 1236 tied 1636 1564 1345 1236 1672 1563 1599 1563 1236 1236 tied 0836 0836 tied 1018 0873 1236 1236 tied 1273 1273 tied 1055 1018 1091 1091 tied 1527 1490 13 times tied 7 times lost 0 times total unique fn went 302 292 20020903 0218 tim_one timtestpy 111 tokenize_word dropped prefix signature faster let caller also repaired bug one place used well conceptual bug anyway code didnt intended changes stats insignificant way fp stats didnt change fn stats shifted one message cases false negative percentages 1091 1091 tied 0945 0945 tied 1200 1236 lost 1454 1454 tied 1491 1491 tied 1091 1091 tied 1091 1127 lost 1236 1236 tied 1636 1636 tied 1382 1345 1636 1672 lost 1599 1599 tied 1236 1236 tied 0836 0836 tied 1018 1018 tied 1236 1236 tied 1273 1273 tied 1055 1055 tied 1091 1091 tied 1527 1527 tied 1 times tied 16 times lost 3 times total unique unchanged 20020902 1930 tim_one timtestpy 110 dont ask helps dont really know skipping long words generating token brief hint much got skipped makes definite improvement fn rate doesnt affect fp rate since experiment said winner im checking left columan right column false positive percentages 0000 0000 tied 0000 0000 tied 0050 0050 tied 0000 0000 tied 0025 0025 tied 0025 0025 tied 0050 0050 tied 0025 0025 tied 0025 0025 tied 0000 0000 tied 0075 0075 tied 0050 0050 tied 0025 0025 tied 0000 0000 tied 0050 0050 tied 0025 0025 tied 0025 0025 tied 0000 0000 tied 0025 0025 tied 0050 0050 tied 0 times tied 20 times lost 0 times total unique fp went 8 8 false negative percentages 1236 1091 1164 0945 1454 1200 1599 1454 1527 1491 1236 1091 1163 1091 1309 1236 1891 1636 1418 1382 1745 1636 1708 1599 1491 1236 0836 0836 tied 1091 1018 1309 1236 1491 1273 1127 1055 1309 1091 1636 1527 19 times tied 1 times lost 0 times total unique fn went 336 302 20020902 1755 tim_one timtestpy 19 comment changes nesting reduction 20020902 1118 tim_one timtestpy 18 fixed outofdate comments made url clumping lumpier distinguishes among first field second field everything else changed tag names email address fields semantically neutral added line tagging add almost pure win beforeandafter fn rates across 20 runs 1418 1236 1309 1164 1636 1454 1854 1599 1745 1527 1418 1236 1381 1163 1418 1309 2109 1891 1491 1418 1854 1745 1890 1708 1818 1491 1055 0836 1164 1091 1599 1309 1600 1491 1127 1127 1164 1309 1781 1636 increased one run variance appears reduced didnt bother compute though beforeandafter fp rates across 20 runs 0000 0000 0000 0000 0075 0050 0000 0000 0025 0025 0050 0025 0075 0050 0025 0025 0025 0025 0025 0000 0100 0075 0050 0050 0025 0025 0000 0000 0075 0050 0025 0025 0025 0025 0000 0000 0075 0025 0100 0050 note 0025 single message really impossible measure improvement fp rate anymore 4000msg ham sets across 20 runs total unique fn fell 353 336 total unique fp fell 13 8 20020902 1006 tim_one timtestpy 17 number changes significant paying attention subject line wrong said clpy ham corpus unusable due mailmaninjected decorations across 20 test runs total unique false positives fell 23 13 total unique false negatives rose 337 353 neither result statistically significant although bet first one would pissed away days trying come realistic model stat sig means 20020901 1722 tim_one classifierpy 17 added comment block hambias experiments theres clearer example trading precision recall favor either expense degree like fiddling knob 20020901 1442 tim_one timtestpy 16 long new comment block summarizing experiments character ngrams bottom line nothing going lot going grahams scheme believe may still place part wordbased tokenizer though 20020901 1005 tim_one classifierpy 16 spamprob never count unique words anymore counting twice gave small benefit unknown_spamprob 02 thats small drag instead 20020901 0733 tim_one rebalpy 13 timtestpy 15 folding case stay read new comments may bad idea languages though refined embeddedurl tagging scheme curious protocol http spamneutral https strong spam indicator surprised 20020901 0647 tim_one classifierpy 15 spamprob removed useless check wordstream isnt empty one thing didnt work since wordstream often iterator even work isnt needed probability empty wordstream gets computed 05 based total absence evidence 20020901 0537 tim_one timtestpy 14 textparts worm around feels like bug msgwalk barry details 20020901 0509 tim_one rebalpy 12 aha staring checkin msg revealed logic bug explains ham directories sometimes remained unbalanced running randomly selected reservoir msg turned spam wasnt pushing toosmall directory stack 20020901 0456 tim_one timtestpy 13 textparts failing weed redundant html cases like multipartalternative textplain multipartrelated texthtml tokenizer also transforms everything lowercase thats accident due simply im testing cant say sure test runs end far looks like bad idea false positive rate 20020901 0452 tim_one rebalpy 11 little script use rebalance ham corpora deleting turns spam another hamreservoir directory thousand randomly selected msgs presumablygood archive arent used scoring training script marches ham corpora directories used gotten big never happens anymore deletes msgs random gotten small plugs holes moving random msgs reservoir 20020901 0325 tim_one classifierpy 14 timtestpy 12 boost unknown_spamprob spam probability assigned words never seen graham used 02 neil schemenauer reported 05 seemed work better tims contentonly tests headers boosting 05 cut false negative rate 13 fp rate increased fps increase wasnt statistically significant also caught 13 spams erroneously classified ham eyeball common sense effect short messages simply arent many highvalue words word prob 05 effect completely ignored spamprob favor word prob differing 05 02 unknown word favors ham expense kicking word prob 02 08 seems dubious face 20020831 1650 tim_one timtestpy 11 driver ive using test runs specific corpus directories useful stuff 20020831 1649 tim_one classifierpy 13 explanation changes pythondev youll find moderator approves msg 20020829 0704 tim_one testerpy 12 classifierpy 12 testerpy repaired comment false_positivenegative_rate functions return percentage eg 10 instead 001 hard get motivated reduce 001 01 wink grahambayesspamprob new optional bool argument true list 15 strongest word probability pairs returned well overall probability find message scored 20020828 1345 montanaro gbayespy 115 ehh actually didnt work well spurious report well pilot error besides tims report suggests simple strsplit may best tokenizer anyway 20020828 1045 montanaro setuppy 11 trivial little setuppy file dont expect people interested makes tad simpler work two files 20020828 1043 montanaro gbayespy 114 add simple trigram tokenizer seems yield best results ive seen far extensively tested 20020828 0810 tim_one testerpy 11 start testing class isnt lot automates much tedium doctest shows already useful things like remembering inputs misclassified 20020827 0645 tim_one mboxcountpy 15 updated stats barry get fiddled output 20020827 0509 bwarsaw splitpy 15 splitnpy 12 _factory return empty string instead none except clauses forloops wont break prematurely mailboxpys base class defines __iter__ raises stopiteration none return 20020827 0455 tim_one gbayespy 113 mboxcountpy 14 whitespace normalization ambiguous tabs snuck mboxcount 20020827 0440 bwarsaw mboxcountpy 13 stats splitting bw good messages unparseable messages 20020827 0423 bwarsaw mboxcountpy 12 _factory use marker object designate good messages unparseable messages reason returning none except clause _factory caused python 221 exit early loop main print statistics number good messages number unparseable messages 20020827 0306 tim_one cleanarch 12 header separator dont bump msg count end 20020824 0142 tim_one gbayespy 112 classifierpy 11 moved interesting code original gbayespy new classifierpy designed clean interface theres reason keep slamming everything one file evergrowing tokenizer stuff probably also split leaving gbayespy pure driver also repaired _test skips checkin left without binding tokenize function 20020824 0117 tim_one splitnpy 11 utility split mbox n random pieces one gulp gives convenient way break giant corpus multiple files used independently across multiple training testing runs important multiple runs different random samples avoid drawing conclusions based accidents single random training corpus algorithm robust similar performance across runs 20020824 0025 montanaro gbayespy 111 allow command line specification tokenize functions run w flag override default tokenize function run w h flag see list tokenize functions adding new tokenizer make docstring short description add keyvalue pair tokenizers dict key user specifies value tokenize function added two new tokenizers tokenize_wordpairs_foldcase tokenize_words_and_pairs obvious either better preexisting functions probably add info pickle indicates tokenizing function used build could default spam detection runs next step drive spamnonspam corpora selecting various tokenizer functions presenting results tabular form 20020823 1310 tim_one gbayespy 110 spamprob commented subtleties clearjunk undid guidos attempt spaceoptimize problem cant delete entries dict thats crawled iteritems suddenly recall materialized list words deleted first time wrote lot better materialize list tobedeleted words materialize entire database dictitems list 20020823 1236 tim_one mboxcountpy 11 utility count display msgs one unix mboxes 20020823 1211 tim_one splitpy 14 open files binary mode else eg 400mb barrys pythonlist corpus vanishes windows also use filewrite instead print latter invents extra newline 20020822 0701 tim_one gbayespy 19 renamed modtime atime better reflect meaning added comment block explain better 20020821 0807 bwarsaw splitpy 13 guido suggests different order positional args 20020821 0737 bwarsaw splitpy 12 get rid 1 2 arguments make positional 20020821 0718 bwarsaw splitpy 11 simple mailbox splitter 20020821 0642 tim_one gbayespy 18 added bunch simple tokenizers originals renamed tokenize_words_foldcase tokenize_5gram_foldcase_wscollapse new ones tokenize_words tokenize_split_foldcase tokenize_split tokenize_5gram tokenize_10gram tokenize_15gram dont expect last word barry test corpus set easy let data tell us pure strategy works best straight character ngrams appealing theyre simplest languageneutral didnt luck weekend size training data trivial 20020821 0508 bwarsaw cleanarch 11 archive cleaner adapted mailman 21b3 version demailmanified 20020821 0444 gvanrossum gbayespy 17 indent repair clearjunk 20020821 0422 gvanrossum gbayespy 16 minor cleanup move identifying comment top clarify bit add author info theres reason _time _heapreplace hidden names change back time heapreplace rename main1 _test main2 main main sees options arguments runs _test get rid list comprehension clearjunk put wordinfoget local variable _add_msg 20020820 1516 tim_one gbayespy 15 neutral typo repairs except clearjunk better chance blowing immediately 20020820 1349 montanaro gbayespy 14 help make easily executable 20020820 0932 bwarsaw gbayespy 13 lots hacks great small main program didnt touch guts algorithm added module docstringusage message added bunch switches train system mbox known good known spam messages using portableunixmailbox uses email package decoding message bodies also allows specify file pickling training data setting threshold messages get xbayesscore header also output messages marked unmarked output file retraining print statistics end 20020820 0543 tim_one gbayespy 12 turned debugging vrbl mistakenly checked true unlearn gave update_probabilitiestrue default arg symmetry learn 20020820 0333 tim_one gbayespy 11 implementation paul grahams bayeslike spam classifier deleted trunkwebsitepresfchangeloght trunkwebsitepresfchangeloght 20070724 000432 utc rev 3154 trunkwebsitepresfchangeloght 20070725 134942 utc rev 3155 1905 00 presourceforge changelog changelog lists commits spambayes projects separate project set see also old cvs repository dont forget date probably want looking current cvs 20020906 0227 tim_one gbayespy 116 testerpy 14 classifierpy 112 cleanarch 13 mboxcountpy 16 rebalpy 14 setuppy 12 splitpy 16 splitnpy 13 timtestpy 118 code moved new sourceforge project spambayes 20020905 1537 tim_one classifierpy 111 added note mincount oddities 20020905 1432 tim_one timtestpy 117 added note word length 20020905 1348 tim_one timtestpy 116 tokenize_word oops awfully permissive took email address tightened also avoided 5graming email addresses w highbit characters false positive percentages 0000 0000 tied 0000 0000 tied 0050 0050 tied 0000 0000 tied 0025 0025 tied 0025 0025 tied 0050 0050 tied 0025 0025 tied 0025 0025 tied 0025 0050 lost 0075 0075 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0000 0000 tied 0025 0025 tied 0050 0050 tied 0 times tied 19 times lost 1 times total unique fp went 7 8 false negative percentages 0764 0691 0691 0655 0981 0945 1309 1309 tied 1418 1164 0873 0800 0800 0763 1163 1163 tied 1491 1345 1200 1127 1381 1345 1454 1490 lost 1164 0909 0655 0582 0655 0691 lost 1163 1163 tied 1200 1018 0982 0873 0982 0909 1236 1127 15 times tied 3 times lost 2 times total unique fn went 260 249 note two losses consist 1 msg difference wins bigger well common 260249 11 spams longer sneak run 4 260 spams used sneak thru 20020905 1151 tim_one classifierpy 110 comment test results moving max_discriminators back 15 doesnt really matter leaving alone 20020905 1002 tim_one classifierpy 19 nowrare pure win changing spamprob work harder find evidence competing 001 099 clues appear left column right false positive percentages 0000 0000 tied 0000 0000 tied 0050 0050 tied 0000 0000 tied 0025 0025 tied 0025 0025 tied 0050 0050 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0075 0075 tied 0025 0025 tied 0025 0025 tied 0025 0025 tied 0075 0025 0025 0025 tied 0025 0025 tied 0000 0000 tied 0025 0025 tied 0050 0050 tied 1 times tied 19 times lost 0 times total unique fp went 9 7 false negative percentages 0909 0764 0800 0691 1091 0981 1381 1309 1491 1418 1055 0873 0945 0800 1236 1163 1564 1491 1200 1200 tied 1454 1381 1599 1454 1236 1164 0800 0655 0836 0655 1236 1163 1236 1200 1055 0982 1127 0982 1381 1236 19 times tied 1 times lost 0 times total unique fn went 284 260 20020904 1121 tim_one timtestpy 115 augmented spam callback display spams low probability 20020904 0953 tim_one testerpy 13 timtestpy 114 added support simple histograms probability distributions ham spam 20020903 1213 tim_one timtestpy 113 reluctant principle change matter stats take stab removing html decorations plain text msgs see comments plain text msgs puts end false positives due text msgs talking html surprisingly also gets rid false negatives surprisingly introduced another small class false positives due dumbass regexp trick used approximate html tag removal removing pieces text nothing html tags eg happened middle uuencoded py file happened leave behind string looked like spam phrase looked like pile long lines didnt generate tokens nonsense outcome either way false positive percentages 0000 0000 tied 0000 0000 tied 0050 0050 tied 0000 0000 tied 0025 0025 tied 0025 0025 tied 0050 0050 tied 0025 0025 tied 0025 0025 tied 0000 0025 lost 0075 0075 tied 0050 0025 0025 0025 tied 0000 0025 lost 0050 0075 lost 0025 0025 tied 0025 0025 tied 0000 0000 tied 0025 0025 tied 0050 0050 tied 1 times tied 16 times lost 3 times total unique fp went 8 9 false negative percentages 0945 0909 0836 0800 1200 1091 1418 1381 1455 1491 lost 1091 1055 1091 0945 1236 1236 tied 1564 1564 tied 1236 1200 1563 1454 1563 1599 lost 1236 1236 tied 0836 0800 0873 0836 1236 1236 tied 1273 1236 1018 1055 lost 1091 1127 lost 1490 1381 12 times tied 4 times lost 4 times total unique fn went 292 284 20020903 0657 tim_one classifierpy 18 added new xspamprob method computes combined probability correctly long comment block explaining happened tried theres something worth pursuing greatly improves false negative rate change alone pushes many marginal hams spam camp 20020903 0523 tim_one timtestpy 112 made skip tokens shorter added surprising treatment organization headers tiny fn benefit tiny cost change fp stats false negative percentages 1091 0945 0945 0836 1236 1200 1454 1418 1491 1455 1091 1091 tied 1127 1091 1236 1236 tied 1636 1564 1345 1236 1672 1563 1599 1563 1236 1236 tied 0836 0836 tied 1018 0873 1236 1236 tied 1273 1273 tied 1055 1018 1091 1091 tied 1527 1490 13 times tied 7 times lost 0 times total unique fn went 302 292 20020903 0218 tim_one timtestpy 111 tokenize_word dropped prefix signature faster let caller also repaired bug one place used well conceptual bug anyway code didnt intended changes stats insignificant way fp stats didnt change fn stats shifted one message cases false negative percentages 1091 1091 tied 0945 0945 tied 1200 1236 lost 1454 1454 tied 1491 1491 tied 1091 1091 tied 1091 1127 lost 1236 1236 tied 1636 1636 tied 1382 1345 1636 1672 lost 1599 1599 tied 1236 1236 tied 0836 0836 tied 1018 1018 tied 1236 1236 tied 1273 1273 tied 1055 1055 tied 1091 1091 tied 1527 1527 tied 1 times tied 16 times lost 3 times total unique unchanged 20020902 1930 tim_one timtestpy 110 dont ask helps dont really know skipping long words generating token brief hint much got skipped makes definite improvement fn rate doesnt affect fp rate since experiment said winner im checking left columan right column false positive percentages 0000 0000 tied 0000 0000 tied 0050 0050 tied 0000 0000 tied 0025 0025 tied 0025 0025 tied 0050 0050 tied 0025 0025 tied 0025 0025 tied 0000 0000 tied 0075 0075 tied 0050 0050 tied 0025 0025 tied 0000 0000 tied 0050 0050 tied 0025 0025 tied 0025 0025 tied 0000 0000 tied 0025 0025 tied 0050 0050 tied 0 times tied 20 times lost 0 times total unique fp went 8 8 false negative percentages 1236 1091 1164 0945 1454 1200 1599 1454 1527 1491 1236 1091 1163 1091 1309 1236 1891 1636 1418 1382 1745 1636 1708 1599 1491 1236 0836 0836 tied 1091 1018 1309 1236 1491 1273 1127 1055 1309 1091 1636 1527 19 times tied 1 times lost 0 times total unique fn went 336 302 20020902 1755 tim_one timtestpy 19 comment changes nesting reduction 20020902 1118 tim_one timtestpy 18 fixed outofdate comments made url clumping lumpier distinguishes among first field second field everything else changed tag names email address fields semantically neutral added line tagging add almost pure win beforeandafter fn rates across 20 runs 1418 1236 1309 1164 1636 1454 1854 1599 1745 1527 1418 1236 1381 1163 1418 1309 2109 1891 1491 1418 1854 1745 1890 1708 1818 1491 1055 0836 1164 1091 1599 1309 1600 1491 1127 1127 1164 1309 1781 1636 increased one run variance appears reduced didnt bother compute though beforeandafter fp rates across 20 runs 0000 0000 0000 0000 0075 0050 0000 0000 0025 0025 0050 0025 0075 0050 0025 0025 0025 0025 0025 0000 0100 0075 0050 0050 0025 0025 0000 0000 0075 0050 0025 0025 0025 0025 0000 0000 0075 0025 0100 0050 note 0025 single message really impossible measure improvement fp rate anymore 4000msg ham sets across 20 runs total unique fn fell 353 336 total unique fp fell 13 8 20020902 1006 tim_one timtestpy 17 number changes significant paying attention subject line wrong said clpy ham corpus unusable due mailmaninjected decorations across 20 test runs total unique false positives fell 23 13 total unique false negatives rose 337 353 neither result statistically significant although bet first one would pissed away days trying come realistic model stat sig means 20020901 1722 tim_one classifierpy 17 added comment block hambias experiments theres clearer example trading precision recall favor either expense degree like fiddling knob 20020901 1442 tim_one timtestpy 16 long new comment block summarizing experiments character ngrams bottom line nothing going lot going grahams scheme believe may still place part wordbased tokenizer though 20020901 1005 tim_one classifierpy 16 spamprob never count unique words anymore counting twice gave small benefit unknown_spamprob 02 thats small drag instead 20020901 0733 tim_one rebalpy 13 timtestpy 15 folding case stay read new comments may bad idea languages though refined embeddedurl tagging scheme curious protocol http spamneutral https strong spam indicator surprised 20020901 0647 tim_one classifierpy 15 spamprob removed useless check wordstream isnt empty one thing didnt work since wordstream often iterator even work isnt needed probability empty wordstream gets computed 05 based total absence evidence 20020901 0537 tim_one timtestpy 14 textparts worm around feels like bug msgwalk barry details 20020901 0509 tim_one rebalpy 12 aha staring checkin msg revealed logic bug explains ham directories sometimes remained unbalanced running randomly selected reservoir msg turned spam wasnt pushing toosmall directory stack 20020901 0456 tim_one timtestpy 13 textparts failing weed redundant html cases like multipartalternative textplain multipartrelated texthtml tokenizer also transforms everything lowercase thats accident due simply im testing cant say sure test runs end far looks like bad idea false positive rate 20020901 0452 tim_one rebalpy 11 little script use rebalance ham corpora deleting turns spam another hamreservoir directory thousand randomly selected msgs presumablygood archive arent used scoring training script marches ham corpora directories used gotten big never happens anymore deletes msgs random gotten small plugs holes moving random msgs reservoir 20020901 0325 tim_one classifierpy 14 timtestpy 12 boost unknown_spamprob spam probability assigned words never seen graham used 02 neil schemenauer reported 05 seemed work better tims contentonly tests headers boosting 05 cut false negative rate 13 fp rate increased fps increase wasnt statistically significant also caught 13 spams erroneously classified ham eyeball common sense effect short messages simply arent many highvalue words word prob 05 effect completely ignored spamprob favor word prob differing 05 02 unknown word favors ham expense kicking word prob 02 08 seems dubious face 20020831 1650 tim_one timtestpy 11 driver ive using test runs specific corpus directories useful stuff 20020831 1649 tim_one classifierpy 13 explanation changes pythondev youll find moderator approves msg 20020829 0704 tim_one testerpy 12 classifierpy 12 testerpy repaired comment false_positivenegative_rate functions return percentage eg 10 instead 001 hard get motivated reduce 001 01 wink grahambayesspamprob new optional bool argument true list 15 strongest word probability pairs returned well overall probability find message scored 20020828 1345 montanaro gbayespy 115 ehh actually didnt work well spurious report well pilot error besides tims report suggests simple strsplit may best tokenizer anyway 20020828 1045 montanaro setuppy 11 trivial little setuppy file dont expect people interested makes tad simpler work two files 20020828 1043 montanaro gbayespy 114 add simple trigram tokenizer seems yield best results ive seen far extensively tested 20020828 0810 tim_one testerpy 11 start testing class isnt lot automates much tedium doctest shows already useful things like remembering inputs misclassified 20020827 0645 tim_one mboxcountpy 15 updated stats barry get fiddled output 20020827 0509 bwarsaw splitpy 15 splitnpy 12 _factory return empty string instead none except clauses forloops wont break prematurely mailboxpys base class defines __iter__ raises stopiteration none return 20020827 0455 tim_one gbayespy 113 mboxcountpy 14 whitespace normalization ambiguous tabs snuck mboxcount 20020827 0440 bwarsaw mboxcountpy 13 stats splitting bw good messages unparseable messages 20020827 0423 bwarsaw mboxcountpy 12 _factory use marker object designate good messages unparseable messages reason returning none except clause _factory caused python 221 exit early loop main print statistics number good messages number unparseable messages 20020827 0306 tim_one cleanarch 12 header separator dont bump msg count end 20020824 0142 tim_one gbayespy 112 classifierpy 11 moved interesting code original gbayespy new classifierpy designed clean interface theres reason keep slamming everything one file evergrowing tokenizer stuff probably also split leaving gbayespy pure driver also repaired _test skips checkin left without binding tokenize function 20020824 0117 tim_one splitnpy 11 utility split mbox n random pieces one gulp gives convenient way break giant corpus multiple files used independently across multiple training testing runs important multiple runs different random samples avoid drawing conclusions based accidents single random training corpus algorithm robust similar performance across runs 20020824 0025 montanaro gbayespy 111 allow command line specification tokenize functions run w flag override default tokenize function run w h flag see list tokenize functions adding new tokenizer make docstring short description add keyvalue pair tokenizers dict key user specifies value tokenize function added two new tokenizers tokenize_wordpairs_foldcase tokenize_words_and_pairs obvious either better preexisting functions probably add info pickle indicates tokenizing function used build could default spam detection runs next step drive spamnonspam corpora selecting various tokenizer functions presenting results tabular form 20020823 1310 tim_one gbayespy 110 spamprob commented subtleties clearjunk undid guidos attempt spaceoptimize problem cant delete entries dict thats crawled iteritems suddenly recall materialized list words deleted first time wrote lot better materialize list tobedeleted words materialize entire database dictitems list 20020823 1236 tim_one mboxcountpy 11 utility count display msgs one unix mboxes 20020823 1211 tim_one splitpy 14 open files binary mode else eg 400mb barrys pythonlist corpus vanishes windows also use filewrite instead print latter invents extra newline 20020822 0701 tim_one gbayespy 19 renamed modtime atime better reflect meaning added comment block explain better 20020821 0807 bwarsaw splitpy 13 guido suggests different order positional args 20020821 0737 bwarsaw splitpy 12 get rid 1 2 arguments make positional 20020821 0718 bwarsaw splitpy 11 simple mailbox splitter 20020821 0642 tim_one gbayespy 18 added bunch simple tokenizers originals renamed tokenize_words_foldcase tokenize_5gram_foldcase_wscollapse new ones tokenize_words tokenize_split_foldcase tokenize_split tokenize_5gram tokenize_10gram tokenize_15gram dont expect last word barry test corpus set easy let data tell us pure strategy works best straight character ngrams appealing theyre simplest languageneutral didnt luck weekend size training data trivial 20020821 0508 bwarsaw cleanarch 11 archive cleaner adapted mailman 21b3 version demailmanified 20020821 0444 gvanrossum gbayespy 17 indent repair clearjunk 20020821 0422 gvanrossum gbayespy 16 minor cleanup move identifying comment top clarify bit add author info theres reason _time _heapreplace hidden names change back time heapreplace rename main1 _test main2 main main sees options arguments runs _test get rid list comprehension clearjunk put wordinfoget local variable _add_msg 20020820 1516 tim_one gbayespy 15 neutral typo repairs except clearjunk better chance blowing immediately 20020820 1349 montanaro gbayespy 14 help make easily executable 20020820 0932 bwarsaw gbayespy 13 lots hacks great small main program didnt touch guts algorithm added module docstringusage message added bunch switches train system mbox known good known spam messages using portableunixmailbox uses email package decoding message bodies also allows specify file pickling training data setting threshold messages get xbayesscore header also output messages marked unmarked output file retraining print statistics end 20020820 0543 tim_one gbayespy 12 turned debugging vrbl mistakenly checked true unlearn gave update_probabilitiestrue default arg symmetry learn 20020820 0333 tim_one gbayespy 11 implementation paul grahams bayeslike spam classifier modified trunkwebsiteunixht trunkwebsiteunixht 20070724 000432 utc rev 3154 trunkwebsiteunixht 20070725 134942 utc rev 3155 98 98 hrefdownloadhtmlrecent enough version python installed install spambayes source either bundled package cvs choose spambayes application best fits package subversion choose spambayes application best fits mail setup procmail 528 528 additional details available hammie readme hrefhttpspambayessvnsourceforgenetviewvccheckoutspambayestrunkspambayesreadmetxtreadme file pop3 2088 2088 training see hammie readme detailed discussion many training options hrefhttpspambayessvnsourceforgenetviewvccheckoutspambayestrunkspambayesreadmetxt readme filefor detailed discussion many training options unix systems notes modified trunkwebsitewindowsht trunkwebsitewindowsht 20070724 000432 utc rev 3154 trunkwebsitewindowsht 20070725 134942 utc rev 3155 357 357 report add useful information may open new bug report problem seems new one please sure go hrefhttpspambayessvnsourceforgenetviewvccheckoutspambayestrunkspambayesoutlook2000docstroubleshootinghtml troubleshootinghtml file installed plugin installing outlook client source 5610 5610 spambayes source either zip file via cvs zip file probably easier handle may improvements code make cvs version viable option though cvs client windows file via subversion zip file probably easier handle may improvements code make subversion version viable option though subversion client windows installed sent sourceforgenet collaborative development platform worlds largest open source development site _______________________________________________ spambayescheckins mailing list pnperxdpvsarswdlppythonorg httpmailpythonorgmailmanlistinfospambayescheckins tue 05 aug 2008 173511 0700 spambayescheckins sfnet svn spambayes 3155 trunkwebsite