skippoboxcom skip montanaro wrote script time ago try minimize duplicates see calculating loose checksum still duplicates delete duplicates training tim people cant stop thinking classifier work tim best trained wholly random spattering real life tim real life contains duplicates thats classifier tim see bit detail get destined many addresses skippoboxcom skipcalendarcom concertsmusicalcom webmastermojamcom etc originally wrote slightly different version loosecksumpy script im check avoid manually scanning presumed spams really identical message identified spam refer loose checksum computed try avoid saving spam multiple times later review would people interested script id happy extricate local modules check cvs tim sure think relevant maybe another purpose paul tim svensson thinking harder real people rest tim us may able get use approaches tim identify closely related spam example amount spam tim going end ham training data real life use tim sort similarity score piece known spam may aid tim finding purging ill check let know find useful skip mon 9 sep 2002 141904 0500 spambayes deleting duplicate spam training good idea