Merge files & keep bigger file if duplicates? I have several folders each containing several hundred thousands of txt-files, let's call them f1, f2, f3, f4.

I want to move all the files into one folder, say F0 and – if there are duplicate files – keep the bigger files (kb or line count) and delete the smaller ones.

Is there a way to do this relatively quickly?

Many thanks in advance!

edit: Upon trying a simple mv-command, I got -bash: /bin/mv: Argument list too long. So I believe that's another hurdle to be overcome.
If you omit the size test

find f1 f2 f3 f4 -type f -exec mv {} f0/ \;


should do the trick.

With a line count (so the file with more lines wins) and assuming that all files are text files it gets a bit more sophisticated:

find f1 f2 f3 f4 -type f -execdir sh -c \
    '[[ ! -e f0/"$1" || $(wc -l &lt; "$1") -gt $(wc -l &lt; f0/"$1") ]] &amp;&amp; \
         mv "$1" f0/' _ {} \;


And finally if you want to compare for file size you need to replace both occurrences of wc -l &lt; (including the &lt;) with stat -f '%z'.

PS: For hundred thousands of text files both find approaches will take a while. The second one using stat should be significantly faster than using wc though.