Hi guys, 

I am trying to use the Dedupe.sh tool from BBTools developed by Brian Bushnell (https://www.biostars.org/u/14684/) to find overlaps in my *de novo* assembled contigs. The input file is a big fasta file containing all contigs generated from different assemblers (MIRA, ABySS, SOAPdenovo, SPAdes), with the sequence headers modified to contain just a sequential counter like >1, >2, ... and so on. I am using the overlap graph generated by Dedupe.sh to merge the contigs in the overlaps using a Perl script. 

The problem is if I try to run Dedupe.sh in the merged assemblies file, I get the following errors during runtime. I made a few edits to the program output pasted below for clarity:

 1. Only showing first 20-something bases of sequences
 2. Separated errors in two different blocks, as the first block of error repeated itself for quite a while

Here is the output of the Dedupe.sh run:

Command: 

    ~/software/bbmap/dedupe.sh in=../../a6_best_assembly_v1.fasta out=a6_best_assembly_v1_DR.fasta outd=a6_best_assembly_v1_duplicates.fasta pattern=a6_best_assembly_v1_cluster% dot=a6_best_assembly_v1_overlapgraph.dot arc am ac fo c mcs=2 pc=t minidentity=95 mo=100 pto=t ngn=f sq=f ple=t overwrite=t

Output:

    java -Djava.library.path=/home/pgbsilva/software/bbmap/jni/ -ea -Xmx68519m -Xms68519m -cp /home/pgbsilva/software/bbmap/current/ jgi.Dedupe in=../../a6_best_assembly_v1.fasta out=a6_best_assembly_v1_DR.fasta outd=a6_best_assembly_v1_duplicates.fasta 
    pattern=a6_best_assembly_v1_cluster% dot=a6_best_assembly_v1_overlapgraph.dot arc am ac fo c mcs=2 pc=t minidentity=95 mo=100 pto=t ngn=f sq=f ple=t overwrite=t
    Executing jgi.Dedupe [in=../../a6_best_assembly_v1.fasta, out=a6_best_assembly_v1_DR.fasta, outd=a6_best_assembly_v1_duplicates.fasta, 
    pattern=a6_best_assembly_v1_cluster%, dot=a6_best_assembly_v1_overlapgraph.dot, arc, am, ac, fo, c, mcs=2, pc=t, minidentity=95, mo=100, pto=t, ngn=f, sq=f, ple=t, overwrite=t]
    
    Initial:
    Memory: max=68853m, free=68135m, used=718m
    
    Found 0 duplicates.
    Finished exact matches.    Time: 0.197 seconds.
    Memory: max=68853m, free=58795m, used=10058m
    
    Found 0 contained sequences.
    Finished containment.      Time: 0.168 seconds.
    Memory: max=68853m, free=63082m, used=5771m
    
    Removed 0 invalid entries.
    Finished invalid removal.  Time: 0.002 seconds.
    Memory: max=68853m, free=63082m, used=5771m
    
**First error block**

    Exception in thread "Thread-72" java.lang.AssertionError: 
    type=FORWARD, len=567, subs=299, edits=0 (175032, length=12932, start1=12365, stop1=12931) (229892, length=1324, start2=0, stop2=566)
    >1
    ATTCCTTGAGTTTTTCTTCCAACCATTTTACTAACATTTTAATTTCTGCTCTCCTATTTTCAGTTATTGAGATTTTTTGCCTGGTGTTTCTGTTTATGGCCTTCTAATTTTGTTCCATGAATGCAATAAGTTCTCCT **(sequence continues)**
    >2
    TTTCTTCACAGAATTGGAAAAAACTACTTTAAAGTTCATATGGAACCAAAAAAGAGCCCGCATTGCCAAGTCAATCCTAAGCCAAAAGAACAAAGCTGGAGGCATCACACTACCTGACTTCAAACTATACTACAAGG **(sequence continues)**
    
    	at jgi.Dedupe$Overlap.<init>(Dedupe.java:3989)
    	at jgi.Dedupe$Unit.makeOverlapReverse(Dedupe.java:5295)
    	at jgi.Dedupe$Unit.makeOverlap(Dedupe.java:4856)
    	at jgi.Dedupe$HashThread.findOverlaps(Dedupe.java:3410)
    	at jgi.Dedupe$HashThread.processRead(Dedupe.java:3274)
    	at jgi.Dedupe$HashThread.processReadOuter(Dedupe.java:3152)
    	at jgi.Dedupe$HashThread.run(Dedupe.java:3085)
    Exception in thread "Thread-63" java.lang.AssertionError: 
    type=FORWARD, len=714, subs=491, edits=0 (175050, length=11792, start1=11078, stop1=11791) (210803, length=1497, start2=0, stop2=713)
    >1
    ACCAGCATATACAGAGACCAAATCAATACAAATAGCAAAGTTAGTATAACATGCTAGTTTTGAAATGATTAATATGTAATATGTTTTTGGAAATTATTAGTTGATTTATTCCTTTACTCACAAATATTTATTCAGT **(sequence continues)**
    >2
    CCGTTTTAGGCGCAACAGACCAACCAGACCAGAATGGATTCATCCATACTAAGTGCCATGTAATCAAACTGACTCATACGGACCAGTTTTCCAAAAAACCTGAAGTAGAATGAAAGGAATATAAAGGAAGATACAG **(sequence continues)**
    
    	at jgi.Dedupe$Overlap.<init>(Dedupe.java:3989)
    	at jgi.Dedupe$Unit.makeOverlapReverse(Dedupe.java:5295)
    	at jgi.Dedupe$Unit.makeOverlap(Dedupe.java:4856)
    	at jgi.Dedupe$HashThread.findOverlaps(Dedupe.java:3410)
    	at jgi.Dedupe$HashThread.processRead(Dedupe.java:3274)
    	at jgi.Dedupe$HashThread.processReadOuter(Dedupe.java:3152)
    	at jgi.Dedupe$HashThread.run(Dedupe.java:3085)
    Exception in thread "Thread-53" java.lang.AssertionError: 
    type=FORWARD, len=263, subs=28, edits=0 (221070, length=1188, start1=0, stop1=262) (223564, length=1064, start2=801, stop2=1063)
    >1
    AAGTGGAGCTGGCTTGGAAAGAATAGGGAAACGGGTGCAACTCCCGTGCGGTTACGCCGCTGTAACAAGTGACGAAGGCTTTATCTATAGCCACTGTCGCACCTGCCTCTTATACACAGCTGACGCTGCCGACGA **(sequence continues)**
    >2
    AGAAGACCTGCTTTTTCATGCTCATCACTCCCATGTAAATCGGGAGACTGTCTCGCTAAAGACAGGATGCTGTCTTTTATACACAGCTGACGCTGCCGACGACGCCTCTAGTTTATTCGTCTGTTGTCGCTCACA **(sequence continues)**
    
    	at jgi.Dedupe$Overlap.<init>(Dedupe.java:3989)
    	at jgi.Dedupe$Unit.makeOverlapReverse(Dedupe.java:5295)
    	at jgi.Dedupe$Unit.makeOverlap(Dedupe.java:4856)
    	at jgi.Dedupe$HashThread.findOverlaps(Dedupe.java:3410)
    	at jgi.Dedupe$HashThread.processRead(Dedupe.java:3274)
    	at jgi.Dedupe$HashThread.processReadOuter(Dedupe.java:3152)
    	at jgi.Dedupe$HashThread.run(Dedupe.java:3085)
    Exception in thread "Thread-69" java.lang.AssertionError: 
    type=FORWARDRC, len=912, subs=77, edits=0 (188, length=1866, start1=954, stop1=1865) (211768, length=1176, start2=1175, stop2=264)
    >1
    TGTCGCTACCGCGATAGGGCAAAAAGTCCTAAAGTTTAGTAAGTGTTTGCTTGGAACACTTTTTCATGAGCCCTTTAATAAGGGGCAGTGGAAGAAATTCATGTAGAGCTCCTTTTTTTTGCATCAATAGGCAA **(sequence continues)**
    >2
    ATAAAGCGAAAGAGAGCGCTTTTTTTTCAGCGTCTAAATTCTTCGTATGATTTCCCTCACATAGTTAGCGAAATCCATTTCCAATGCACTGCATTTGGAAATTTTTTGCCTATTGATGCAAAAAAAAGGAGCTC **(sequence continues)**
    
    	at jgi.Dedupe$Overlap.<init>(Dedupe.java:3989)
    	at jgi.Dedupe$Unit.makeOverlapForwardRC(Dedupe.java:5229)
    	at jgi.Dedupe$Unit.makeOverlap(Dedupe.java:4851)
    	at jgi.Dedupe$HashThread.findOverlaps(Dedupe.java:3410)
    	at jgi.Dedupe$HashThread.processRead(Dedupe.java:3274)
    	at jgi.Dedupe$HashThread.processReadOuter(Dedupe.java:3152)
    	at jgi.Dedupe$HashThread.run(Dedupe.java:3085)
    Exception in thread "Thread-59" java.lang.AssertionError: 
    type=FORWARD, len=623, subs=363, edits=0 (174723, length=12656, start1=12033, stop1=12655) (20470, length=9394, start2=0, stop2=622)
    >1
    GTTTGCGAAACTAAAGACAAAAGAAATGCCATAAAAATATCTTCTAGATGACAAAGTTGTGCCTTTTGGAGTTGCATTTTAACACATCGAAACCACTACACACATACACGGGAACTGCACAATTGGGTAAATA **(sequence continues)**
    >2
    ATGTGCAAGTTTGTTACATGGGTATACATGTGCTATGTTGGTTTGTTGCACCTATTAACTCATCACTTACATTGGGTATTTCTCCTAATGCTATCCTTCCTCCAGCCCCCCACCCCATGACAGGCCCCAGTGT **(sequence continues)**
    
    	at jgi.Dedupe$Overlap.<init>(Dedupe.java:3989)
    	at jgi.Dedupe$Unit.makeOverlapReverse(Dedupe.java:5295)
    	at jgi.Dedupe$Unit.makeOverlap(Dedupe.java:4856)
    	at jgi.Dedupe$HashThread.findOverlaps(Dedupe.java:3410)
    	at jgi.Dedupe$HashThread.processRead(Dedupe.java:3274)
    	at jgi.Dedupe$HashThread.processReadOuter(Dedupe.java:3152)
    	at jgi.Dedupe$HashThread.run(Dedupe.java:3085)
    Exception in thread "Thread-57" java.lang.AssertionError: 
    type=FORWARD, len=210, subs=15, edits=0 (66196, length=2881, start1=2671, stop1=2880) (67503, length=1435, start2=0, stop2=209)
    >1
    GCTCTTTGGAATGCCAGACGCAGTGGCATGTACCTGTAGACCCACCTACAAGGTGGGCTGTTGTGGGCTGTAGTGTGCTGTGATTTTGCCTGTGATACCCACTGCCCTCCAGCCTAGGCAACATAGTGAGA **(sequence continues)**
    >2
    GCCACAGTTTCTTAATCCAGTCTATCACTGATGGACATTTGGGTTGGTTCCAAGTCTTTGCTATTGTGAATAGTGCCGCAATAAACATACGTGTGCATGTGTCTTTATAGCAGCATGATTTATAATCCTTT **(sequence continues)**
    
    	at jgi.Dedupe$Overlap.<init>(Dedupe.java:3989)
    	at jgi.Dedupe$Unit.makeOverlapReverse(Dedupe.java:5295)
    	at jgi.Dedupe$Unit.makeOverlap(Dedupe.java:4856)
    	at jgi.Dedupe$HashThread.findOverlaps(Dedupe.java:3410)
    	at jgi.Dedupe$HashThread.processRead(Dedupe.java:3274)
    	at jgi.Dedupe$HashThread.processReadOuter(Dedupe.java:3152)
    	at jgi.Dedupe$HashThread.run(Dedupe.java:3085)
    	
    	
This set of errors continues for a few more times. The program continues running and outputs this:
    
    Found 241 overlaps.
    Finished finding overlaps. Time: 0.109 seconds.
    Memory: max=68853m, free=62391m, used=6462m
    
    Overlaps:       540,  	length: 1025664
    Counted overlaps.          Time: 0.003 seconds.
    Memory: max=68853m, free=62391m, used=6462m
    
    Clusters:         8104 (114 of at least size 2)
    
    Size Range        Clusters          Reads             Bases
    1                 7990              7990              24600288
    2                 55                110               562883
    3-4               42                145               665875
    5-8               15                91                432193
    9-16              1                 15                48450
    17-32             1                 17                128562
    
    Largest:          17
    Finished making clusters.  Time: 0.012 seconds.
    Memory: max=68853m, free=62391m, used=6462m
    
    Removed 0 invalid entries.
    Finished invalid removal.  Time: 0.001 seconds.
    Memory: max=68853m, free=62391m, used=6462m
    
**Second set of errors:**

    Exception in thread "Thread-79" java.lang.AssertionError
    	at jgi.Dedupe$Overlap.flip(Dedupe.java:4126)
    	at jgi.Dedupe$ClusterThread.canonicize(Dedupe.java:2858)
    	at jgi.Dedupe$ClusterThread.canonicizeNeighbors(Dedupe.java:2723)
    	at jgi.Dedupe$ClusterThread.canonicizeClusterBreadthFirst(Dedupe.java:2660)
    	at jgi.Dedupe$ClusterThread.run(Dedupe.java:2080)
    Exception in thread "Thread-91" java.lang.AssertionError
    	at jgi.Dedupe$Overlap.flip(Dedupe.java:4126)
    	at jgi.Dedupe$ClusterThread.canonicize(Dedupe.java:2858)
    	at jgi.Dedupe$ClusterThread.canonicizeNeighbors(Dedupe.java:2723)
    	at jgi.Dedupe$ClusterThread.canonicizeClusterBreadthFirst(Dedupe.java:2660)
    	at jgi.Dedupe$ClusterThread.run(Dedupe.java:2080)
    Exception in thread "Thread-90" Exception in thread "Thread-78" java.lang.AssertionError
    	at jgi.Dedupe$Overlap.flip(Dedupe.java:4126)
    	at jgi.Dedupe$ClusterThread.canonicize(Dedupe.java:2858)
    	at jgi.Dedupe$ClusterThread.canonicizeNeighbors(Dedupe.java:2723)
    	at jgi.Dedupe$ClusterThread.canonicizeClusterBreadthFirst(Dedupe.java:2660)
    	at jgi.Dedupe$ClusterThread.run(Dedupe.java:2080)
    java.lang.AssertionError
    	at jgi.Dedupe$Overlap.flip(Dedupe.java:4126)
    	at jgi.Dedupe$ClusterThread.canonicize(Dedupe.java:2858)
    	at jgi.Dedupe$ClusterThread.canonicizeNeighbors(Dedupe.java:2723)
    	at jgi.Dedupe$ClusterThread.canonicizeClusterBreadthFirst(Dedupe.java:2660)
    	at jgi.Dedupe$ClusterThread.run(Dedupe.java:2080)
    Exception in thread "Thread-98" java.lang.AssertionError
    	at jgi.Dedupe$ClusterThread.canonicize(Dedupe.java:2864)
    	at jgi.Dedupe$ClusterThread.canonicizeNeighbors(Dedupe.java:2723)
    	at jgi.Dedupe$ClusterThread.canonicizeClusterBreadthFirst(Dedupe.java:2660)
    	at jgi.Dedupe$ClusterThread.run(Dedupe.java:2080)
    Exception in thread "Thread-81" java.lang.AssertionError
    	at jgi.Dedupe$Overlap.flip(Dedupe.java:4126)
    	at jgi.Dedupe$ClusterThread.canonicize(Dedupe.java:2858)
    	at jgi.Dedupe$ClusterThread.canonicizeNeighbors(Dedupe.java:2723)
    	at jgi.Dedupe$ClusterThread.canonicizeClusterBreadthFirst(Dedupe.java:2660)
    	at jgi.Dedupe$ClusterThread.run(Dedupe.java:2080)

The second block of error also repeats a few more times.

    Found 6 multijoins (4178 bases).
    Experienced 0 multijoin removal failures.
    Flipped 118 reads and 162 overlaps.
    Found 0 clusters (0 overlaps) with contradictory orientation cycles.
    Found 1 clusters (2 overlaps) with remaining cycles.
    
    After processing clusters:
    Clusters:         8019 (29 of at least size 2)
    
    Size Range        Clusters          Reads             Bases
    1                 7990              7990              24600288
    2                 29                79                377770
    
    Largest:          7
    Finished processing.       Time: 0.016 seconds.
    Memory: max=68853m, free=61728m, used=7125m
    
    Input:                  	8368 reads 		26438251 bases.
    Duplicates:             	0 reads (0.00%) 	0 bases (0.00%)     	0 collisions.
    Containments:           	0 reads (0.00%) 	0 bases (0.00%)    	121936 collisions.
    Overlaps:               	241 reads (2.88%) 	512832 bases (1.94%)    	1492 collisions.
    Result:                 	8368 reads (100.00%) 	26438251 bases (100.00%)
    
    Printed output.            Time: 0.161 seconds.
    Memory: max=68853m, free=61673m, used=7180m
    
    Time:   			0.685 seconds.
    Reads Processed:        8368 	12.22k reads/sec
    Bases Processed:      26438k 	38.60m bases/sec

I am not experienced in Java so I have no idea what the errors mean but it seems like that for some of the threads, something happened during overlap detection between two contigs and that is causing the error. For the second set, the error happens during the canonization step of Dedupe.sh. I don't know if it happens because of the first one or if it is an unrelated incident. Also, every time I run the program with this dataset, it generates a different number of clusters. 

Does any of you encountered these errors while running Dedupe.sh? Any help would be fantastic!
Thanks for the detailed error report.  I have a couple of ideas about this, but it might be a little difficult to test.  You're getting different (and incorrect) output each time because it's crashing; it yields deterministic output when there are no crashes.

First - can you try running with adding the flag "-da" and see what happens?

Second - it's likely that "minidentity=95" is the cause of the instability.  Can you try running without that flag and see what happens?  If so, there may be some ways to work around it.