This isn't as much of a specific problem as something I am looking for more of a "Pythonic" philosophical answer to.  Namely, what's the best way to keep track of unique items and ensure duplicates don't arise?
For example, I am writing a script to scrape a website for links to songs on SoundCloud so I can automatically download them.  If I want to automate this program with, say, cron, what's the most efficient way to ensure that I am downloading only content I don't have already?
Or if I downloaded images, how could I make sure that there aren't any duplicates, or have some sort of process that searches for and removes duplicates efficiently?
Kind of open ended, so contribute as little or as much as you please.
Thanks.
Use a dict or set.  Consider computing a checksum of each item.  This brings you toward what's known as Content Addressable Storage, which is where the checksum actually is stored as if it were the item's "name", and a separate index is stored which maps things like filenames or song names to the checksums or data blocks.  The problem with the CAS approach in your particular case is that it may not be possible for you to get a checksum computed on the remote side for new content--that's how programs like rsync avoid copying duplicate data.
What is the most efficient way to keep track of unique downloaded files in a Python program?