Body: <p>I have a large-ish pgn composed from multiple sources - logically it has a lot of duplicate games.</p>
  <p>I tried to filter it using <a href="https://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/help.html#duplicates" rel="nofollow noreferrer">pgn-extract duplicate detection</a>:</p>
  <pre><code>pgn-extract --duplicates dupes.pgn --output unique.pgn file.pgn
  </code></pre>
  <p>My problem is, that duplicate detection only compares movetext and no headers - I would like to keep games that have different players, even if they had same moves played.</p>
  <p>For example, using command as written these two games are considered same:</p>
  <pre><code>[White &quot;Some white player&quot;]
  [Black &quot;Some black player&quot;]
  [Result &quot;1/2-1/2&quot;]
   
  1. e4 e5 2. Nf3 Nc6 3. Bb5  1/2-1/2
  </code></pre>
  <pre><code>[White &quot;Some different white player&quot;]
  [Black &quot;Some different black player&quot;]
  [Result &quot;1/2-1/2&quot;]
   
  1. e4 e5 2. Nf3 Nc6 3. Bb5  1/2-1/2
  </code></pre>
  <p>While I would like to consider them different. So which options do I need to use with pgn-extract to compare both movetext and headers when detecting duplicates?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-06T20:25:18.010Z
Comments.Text: I am afraid that that is not possible with pgn-extract. One issue that would need to be addressed is how to specify the tag-matching criteria, given that games often have different sets of tags, and - depending on the sources of the games - variations in the text of tags.
Comments.UserDisplayName: kentdjb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-07T09:04:01.150Z
Comments.Text: Understandable, thank you.
Comments.UserDisplayName: Matija Sirk
Tags: <pgn-extract>
Title: How to make pgn-extract duplicate detection consider headers?
ThreadText: # How to make pgn-extract duplicate detection consider headers?
  
  (Asked by: username_0 on 2024-06-06)
  
  I have a large-ish pgn composed from multiple sources - logically it has a lot of duplicate games.
  
  I tried to filter it using [pgn-extract duplicate detection](https://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/help.html#duplicates):
  
  ```
  pgn-extract --duplicates dupes.pgn --output unique.pgn file.pgn
  
  ```
  
  My problem is, that duplicate detection only compares movetext and no headers - I would like to keep games that have different players, even if they had same moves played.
  
  For example, using command as written these two games are considered same:
  
  ```
  [White "Some white player"]
  [Black "Some black player"]
  [Result "1/2-1/2"]
   
  1. e4 e5 2. Nf3 Nc6 3. Bb5  1/2-1/2
  
  ```
  
  ```
  [White "Some different white player"]
  [Black "Some different black player"]
  [Result "1/2-1/2"]
   
  1. e4 e5 2. Nf3 Nc6 3. Bb5  1/2-1/2
  
  ```
  
  While I would like to consider them different. So which options do I need to use with pgn-extract to compare both movetext and headers when detecting duplicates?
  
  **Tags:** pgn-extract
  
  > **Comments:**
  > **username_1**: I am afraid that that is not possible with pgn-extract. One issue that would need to be addressed is how to specify the tag-matching criteria, given that games often have different sets of tags, and - depending on the sources of the games - variations in the text of tags.  
  > **username_0**: Understandable, thank you.  
  
  ---
  
  ## 0 answers
  
  