nealewoozleorg neale pickett tim peters like tim tests train 7000 msgs binary pickle database approaching 10 million bytes shrinks 2 million bytes though delete wordinfo records spamprob exactly equal unknown_spamprob records arent needed scoring unknown word gets madeup probability unknown_spamprob records needed training ive noted scoringonly database leaner thats pretty good wonder much better could using custom pickler checked little dbm file found lot would call bloat import anydbm hammie hammiepersistentgrahambayeshamdb db anydbmopenhamdb dbneale lendbneale ccopy_regn_reconstructornqx01cclassifiernwordinfonqx02c__builtin__nobjectnqx03ntrqx04gaxcexbcxfdx94xbbokx00kx00kx00gxe0x00x00x00x00x00x00tb 106 dwordinfoneale lendwordinfoneale wordinfo103133797916197 0 0 0 05 42 ignoring fact many zeros pickled version wordinfo object twice large string representation could get 50 decrease size using string representation instead pickle right something logic seems wrong cant see maybe pickling good heterogeneous data types every value big dictionary going type theres ton redundancy guess explains compressed well neale 06 sep 2002 144433 0700 spambayes deployment