nealewoozleorg neale pickett tim peters like guido dont know big pickle would maybe loading time fine maybe marshalling tests train 7000 msgs binary pickle database approaching 10 million bytes paltry 3000message training set makes 63mb 1mb1e6 bytes pickle hammiepy checked optionally let write stuff dbm file message base dbm file weighs hefty 214mb also takes longer write using database real 8m24741s user 6m19410s sys 1m33650s using pickle real 1m39824s user 1m36400s sys 0m2160s piii 551257mhz dont know supposed 551257 proccpuinfo says comparison spamoracle currently gold standard mind least speed data blazes along real 0m29592s user 0m28050s sys 0m1180s data file appears marshalled hash 448kb however compiled ocaml uses much simpler tokenizing algorithm written lexical analyzer ocamllex well never able outperform something keep mind though dont statistics yet scanning unknown messages actually database blows pickle water scores every word 000 im sure thats fair test case 21mb per user probably large 10mb questionable hand pickle compressed well gzip shrinking 18mb neale 06 sep 2002 125833 0700 spambayes deployment