one simple option might be to just use sha256 sums on files. if done incrementally, it&#x27;s not that expensive computationally, unless files are very large (couple of gb&#x27;s).<p>one way to mitigate that might be take a _few_ random, say, 1k chunks of file content, and hash those. as long as you have a map of (offset, hash) de-duping should be just fine.<p>the only thing is, i don&#x27;t really know, what an upper bound on &#x27;few samples&#x27; would be like. asymptotically, it should approach file-size&#x2F;chunk-size, but ideally, it should be way less than that.<p>any ideas on how to determine a minimal but complete set ?