Answers.Body: <p>You may use the more versatile subcommand <code>jja merge</code> of the tool <a href="https://crates.io/crates/jja" rel="nofollow noreferrer">jja</a>:</p>
  <pre><code>⇒  jja merge -h
  Merge two opening books
  
  Usage: jja merge [OPTIONS] --input1 &lt;input-file-1&gt; --input2 &lt;input-file-2&gt; --output &lt;output-file&gt;
  
  Options:
    -1, --input1 &lt;input-file-1&gt;
            First opening book
        --porcelain[=&lt;FORMAT&gt;]
            Give the output in an easy-to-parse format for scripts [default: auto] [possible values: auto, csv]
    -2, --input2 &lt;input-file-2&gt;
            Second opening book
    -v, --verbose...
            Run verbosely, may be specified multiple times [env: JJA_VERBOSITY=]
    -o, --output &lt;output-file&gt;
            Path to the output chess file
    -T, --threads &lt;threads&gt;
            Specify maximum number of threads [env: JJA_NPROC=] [default: 4]
    -s, --strategy &lt;merge-strategy&gt;
            Merge strategy [default: sum] [possible values: avg, max, min, ours, sum, wavg]
    -w, --weight1 &lt;merge-book1-weight&gt;
            Book 1 weight for merge strategy weighted average, aka `wavg'
    -W, --weight2 &lt;merge-book2-weight&gt;
            Book 2 weight for merge strategy weighted average, aka `wavg'
    -c, --weight-cutoff &lt;merge-weight-cutoff&gt;
            Moves less than this weight will not be included in the book
    -h, --help
            Print help
    -V, --version
            Print version
  </code></pre>
  <p>To solve your problem you should use a merge strategy which suits your needs. If you want to keep both entries you would go for either <code>avg</code>, <code>sum</code>, or <code>wavg</code>, <code>max</code> and <code>min</code> which states how the weight field should be updated for entries with same moves with averaging them, summing them and weighted averaging, using the maximum weight, and using the minimum weight being five options.</p>
  <p>If you want to keep a single field, then you should use <code>ours</code> strategy which keeps entries in the first book intact when both book have the same position. I use this very often when I merge a small book with my own openings over a huge book generated from a database of lichess games so I get to use the information from lichess database whilst still having my own preferred moves selected for the positions I want. Another common option I use is <code>--weight-cutoff</code> with which I can get rid off very weak moves in the opening book.</p>
  <p>Finally I should admit I am the author of <a href="https://git.sr.ht/%7Ealip/jja" rel="nofollow noreferrer">jja</a> so this is in-part self-promotion for which I apologize.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-02T23:46:03.010Z
Answers.LastActivityDate: 2023-07-02T23:46:03.010Z
Answers.UserDisplayName: alip
Body: <p>I have been trying to merge polyglot chess opening databases.
  When merging databases with existing programs, if any move in a position already exist in the first file, new moves from same position will not be added from the next file.</p>
  <p>Here is the problem:
  When two books (in bin-format) are merged with polyglot, it's expected
  that lines from both of them can be found in the new combined/merged
  one. This is not the case: the contents of the second entry are omitted:</p>
  <p>pgn1:</p>
  <pre><code>[Event &quot;?&quot;]
  [Site &quot;?&quot;]
  [Date &quot;2021.01.01&quot;]
  [Number &quot;1&quot;]
  [White &quot;?&quot;]
  [Black &quot;?&quot;]
  [Result &quot;1/2-1/2&quot;]
  [ECO &quot;B74&quot;]
  [PlyCount &quot;12&quot;]
  
  1. e4 c5 1/2-1/2
  </code></pre>
  <p>pgn2:</p>
  <pre><code>[Event &quot;?&quot;]
  [Site &quot;?&quot;]
  [Date &quot;2021.01.01&quot;]
  [Number &quot;1&quot;]
  [White &quot;?&quot;]
  [Black &quot;?&quot;]
  [Result &quot;1/2-1/2&quot;]
  [ECO &quot;A00&quot;]
  [PlyCount &quot;12&quot;]
  
  1. g3 e5  1/2-1/2
  </code></pre>
  <p>After doing:</p>
  <pre><code>polyglot make-book -pgn 1.pgn -min-game 1 -max-ply 12 -bin 1.bin
  
  polyglot make-book -pgn 2.pgn -min-game 1 -max-ply 12 -bin 2.bin
  
  polyglot merge-book -in1 1.bin -in2 2.bin -out combined.bin
  </code></pre>
  <p>the <code>g3 e5</code> entry gets scraped.</p>
  <p>For reference, the above definition of the problem was described here:
  <a href="https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1798406.html" rel="nofollow noreferrer">https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1798406.html</a></p>
  <p>This is something I have found really annoying for a very long time.
  After years of frustration that there was no program that could properly merge chess polyglot database files with recalculation of weights, I eventually decided to learn how to make a program that could do that.</p>
  <p>So, now I have written a program to merge chess opening polyglot database files with recalculation of weights and it solves this problem. I call it merge_polyglot.</p>
  <p>But I don´t really have any clue what to do with it. Would others find this useful? Where would I post this program to make it available for other chess enthusiasts?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-06T13:35:27.373Z
Comments.Text: Updated with details of problem description included.
Comments.UserDisplayName: Mats Backstrom
Tags: <software><polyglot>
Title: Program to merge chess opening polyglot database files with recalculation of weights
ThreadText: # Program to merge chess opening polyglot database files with recalculation of weights
  
  (Asked by: username_0 on 2023-02-05)
  
  I have been trying to merge polyglot chess opening databases.
  When merging databases with existing programs, if any move in a position already exist in the first file, new moves from same position will not be added from the next file.
  
  Here is the problem:
  When two books (in bin-format) are merged with polyglot, it's expected
  that lines from both of them can be found in the new combined/merged
  one. This is not the case: the contents of the second entry are omitted:
  
  pgn1:
  
  ```
  [Event "?"]
  [Site "?"]
  [Date "2021.01.01"]
  [Number "1"]
  [White "?"]
  [Black "?"]
  [Result "1/2-1/2"]
  [ECO "B74"]
  [PlyCount "12"]
  
  1. e4 c5 1/2-1/2
  
  ```
  
  pgn2:
  
  ```
  [Event "?"]
  [Site "?"]
  [Date "2021.01.01"]
  [Number "1"]
  [White "?"]
  [Black "?"]
  [Result "1/2-1/2"]
  [ECO "A00"]
  [PlyCount "12"]
  
  1. g3 e5  1/2-1/2
  
  ```
  
  After doing:
  
  ```
  polyglot make-book -pgn 1.pgn -min-game 1 -max-ply 12 -bin 1.bin
  
  polyglot make-book -pgn 2.pgn -min-game 1 -max-ply 12 -bin 2.bin
  
  polyglot merge-book -in1 1.bin -in2 2.bin -out combined.bin
  
  ```
  
  the `g3 e5` entry gets scraped.
  
  For reference, the above definition of the problem was described here:
  <https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1798406.html>
  
  This is something I have found really annoying for a very long time.
  After years of frustration that there was no program that could properly merge chess polyglot database files with recalculation of weights, I eventually decided to learn how to make a program that could do that.
  
  So, now I have written a program to merge chess opening polyglot database files with recalculation of weights and it solves this problem. I call it merge\_polyglot.
  
  But I don´t really have any clue what to do with it. Would others find this useful? Where would I post this program to make it available for other chess enthusiasts?
  
  **Tags:** software, polyglot
  
  > **Comments:**
  > **username_0**: Updated with details of problem description included.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-02. Score: 1
  
  You may use the more versatile subcommand `jja merge` of the tool [jja](https://crates.io/crates/jja):
  
  ```
  ⇒  jja merge -h
  Merge two opening books
  
  Usage: jja merge [OPTIONS] --input1 <input-file-1> --input2 <input-file-2> --output <output-file>
  
  Options:
    -1, --input1 <input-file-1>
            First opening book
        --porcelain[=<FORMAT>]
            Give the output in an easy-to-parse format for scripts [default: auto] [possible values: auto, csv]
    -2, --input2 <input-file-2>
            Second opening book
    -v, --verbose...
            Run verbosely, may be specified multiple times [env: JJA_VERBOSITY=]
    -o, --output <output-file>
            Path to the output chess file
    -T, --threads <threads>
            Specify maximum number of threads [env: JJA_NPROC=] [default: 4]
    -s, --strategy <merge-strategy>
            Merge strategy [default: sum] [possible values: avg, max, min, ours, sum, wavg]
    -w, --weight1 <merge-book1-weight>
            Book 1 weight for merge strategy weighted average, aka `wavg'
    -W, --weight2 <merge-book2-weight>
            Book 2 weight for merge strategy weighted average, aka `wavg'
    -c, --weight-cutoff <merge-weight-cutoff>
            Moves less than this weight will not be included in the book
    -h, --help
            Print help
    -V, --version
            Print version
  
  ```
  
  To solve your problem you should use a merge strategy which suits your needs. If you want to keep both entries you would go for either `avg`, `sum`, or `wavg`, `max` and `min` which states how the weight field should be updated for entries with same moves with averaging them, summing them and weighted averaging, using the maximum weight, and using the minimum weight being five options.
  
  If you want to keep a single field, then you should use `ours` strategy which keeps entries in the first book intact when both book have the same position. I use this very often when I merge a small book with my own openings over a huge book generated from a database of lichess games so I get to use the information from lichess database whilst still having my own preferred moves selected for the positions I want. Another common option I use is `--weight-cutoff` with which I can get rid off very weak moves in the opening book.
  
  Finally I should admit I am the author of [jja](https://git.sr.ht/%7Ealip/jja) so this is in-part self-promotion for which I apologize.
  
  
  ---
  
  