Answers.Body: <p>You can do this with the <code>sort</code> command-line tool. This command pipe sequence sorts numerically by the second field, then chooses unique elements based on the first field</p>
  <pre><code>sort -r -k 2n,2 read_lengths.txt | sort -r -u -k 1,1 &gt; unique_lengths.txt
  </code></pre>
  <p><em>[the additional <code>-r</code> bits are not strictly necessary; they help make sure that the headings stay at the top]</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-11T22:12:41.737Z
Answers.LastActivityDate: 2023-07-12T16:16:09.527Z
Answers.LastEditDate: 2023-07-12T16:16:09.527Z
Answers.UserDisplayName: gringer
Answers.Body: <p><s>Not sure how you'd do this in python but</s> you need a group-by operation here. Group by the read name and pick the min length per group.</p>
  <p>EDIT: With pandas, you can do:</p>
  <pre class="lang-py prettyprint-override"><code>df.groupby(['read'])['length'].agg('min')
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-11T18:45:50.543Z
Answers.Text: That's so much simpler and it worked! Thank you!
Answers.UserDisplayName: rimo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-11T17:48:56.983Z
Answers.LastActivityDate: 2023-07-11T17:48:56.983Z
Answers.UserDisplayName: Ram RS
Body: <p>I have a tsv file that lists the reads and read lengths from a FASTA file but some reads are duplicated - that's just from the analysis I did previously - but I want to only take one instance of the read and have it be the SHORTER length than the other duplicate.</p>
  <p>For instance, here is the beginning of the tsv file:</p>
  <pre><code>read                                    length
  91290b97-5ca1-489d-b488-17afc1d217e7    3584
  dad08436-3cd6-40a1-9af1-b846bd24a815    3787
  f4a7bbdf-de62-48a8-a8b0-06076e251820    3637
  f4a7bbdf-de62-48a8-a8b0-06076e251820    4717
  21ddb2f2-654e-4124-8226-91c5a8d11a97    3786
  6a2de460-a661-4c9e-ac1d-adfc3e5841e2    3520
  </code></pre>
  <p>You can see that <code>f4a7bbdf-de62-48a8-a8b0-06076e251820</code> is found twice with different lengths <code>3637</code> and <code>4717</code>. I want to go through and take the shorter length of that read and include it in the final tsv file, resulting in this structure:</p>
  <pre><code>read                                    length
  91290b97-5ca1-489d-b488-17afc1d217e7    3584
  dad08436-3cd6-40a1-9af1-b846bd24a815    3787
  f4a7bbdf-de62-48a8-a8b0-06076e251820    3637
  21ddb2f2-654e-4124-8226-91c5a8d11a97    3786
  6a2de460-a661-4c9e-ac1d-adfc3e5841e2    3520
  </code></pre>
  <p>I'm trying to do this in python and have started with making the tsv file a list:</p>
  <pre class="lang-py prettyprint-override"><code>newlist = []
  duplist = []
  
  with open(tsv_path) as file:
      tsv_file = csv.reader(file, delimiter=&quot;\t&quot;)
      
      for line in tsv_file:
          if line[0] not in newlist:
              newlist.append(line[0])
          else:
              duplist.append(line)
              
      print(&quot;All the duplicate reads: &quot;, duplist)
      print(&quot;The normal reads: &quot;, newlist)
  </code></pre>
  <p>Which works to contain the first instance of a read in one list and the second/third instance of the read to go in a separate list with their lengths. However, the shorter length of a read is not always the first instance of a read. <strong>Do I need to include another loop that looks at the lengths and compares each instance of a read? Is there an easier way to do this?</strong></p>
  
Tags: <python><file-formats><pandas><duplications>
Title: Getting rid of duplicates in a dictionary
ThreadText: # Getting rid of duplicates in a dictionary
  
  (Asked by: username_0 on 2023-07-11)
  
  I have a tsv file that lists the reads and read lengths from a FASTA file but some reads are duplicated - that's just from the analysis I did previously - but I want to only take one instance of the read and have it be the SHORTER length than the other duplicate.
  
  For instance, here is the beginning of the tsv file:
  
  ```
  read                                    length
  91290b97-5ca1-489d-b488-17afc1d217e7    3584
  dad08436-3cd6-40a1-9af1-b846bd24a815    3787
  f4a7bbdf-de62-48a8-a8b0-06076e251820    3637
  f4a7bbdf-de62-48a8-a8b0-06076e251820    4717
  21ddb2f2-654e-4124-8226-91c5a8d11a97    3786
  6a2de460-a661-4c9e-ac1d-adfc3e5841e2    3520
  
  ```
  
  You can see that `f4a7bbdf-de62-48a8-a8b0-06076e251820` is found twice with different lengths `3637` and `4717`. I want to go through and take the shorter length of that read and include it in the final tsv file, resulting in this structure:
  
  ```
  read                                    length
  91290b97-5ca1-489d-b488-17afc1d217e7    3584
  dad08436-3cd6-40a1-9af1-b846bd24a815    3787
  f4a7bbdf-de62-48a8-a8b0-06076e251820    3637
  21ddb2f2-654e-4124-8226-91c5a8d11a97    3786
  6a2de460-a661-4c9e-ac1d-adfc3e5841e2    3520
  
  ```
  
  I'm trying to do this in python and have started with making the tsv file a list:
  
  ```py
  newlist = []
  duplist = []
  
  with open(tsv_path) as file:
      tsv_file = csv.reader(file, delimiter="\t")
      
      for line in tsv_file:
          if line[0] not in newlist:
              newlist.append(line[0])
          else:
              duplist.append(line)
              
      print("All the duplicate reads: ", duplist)
      print("The normal reads: ", newlist)
  
  ```
  
  Which works to contain the first instance of a read in one list and the second/third instance of the read to go in a separate list with their lengths. However, the shorter length of a read is not always the first instance of a read. **Do I need to include another loop that looks at the lengths and compares each instance of a read? Is there an easier way to do this?**
  
  **Tags:** python, file-formats, pandas, duplications
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-07-11. Score: 3 | Accepted answer
  
  ~~Not sure how you'd do this in python but~~ you need a group-by operation here. Group by the read name and pick the min length per group.
  
  EDIT: With pandas, you can do:
  
  ```py
  df.groupby(['read'])['length'].agg('min')
  
  ```
  
  > **Comments:**
  > **username_0**: That's so much simpler and it worked! Thank you!  
  
  ---
  
  ### Answer by username_1 on 2023-07-11. Score: 2
  
  You can do this with the `sort` command-line tool. This command pipe sequence sorts numerically by the second field, then chooses unique elements based on the first field
  
  ```
  sort -r -k 2n,2 read_lengths.txt | sort -r -u -k 1,1 > unique_lengths.txt
  
  ```
  
  *[the additional `-r` bits are not strictly necessary; they help make sure that the headings stay at the top]*
  
  
  ---
  
  