<p dir="auto">The following looks like a bug to me (but maybe I misunderstand the intent of the two filters in question and how they are meant to interact).</p>
<table role="table">
<thead>
<tr>
<th><strong>Elasticsearch version</strong></th>
<th>2.2</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Plugins installed</strong></td>
<td><a href="https://github.com/francesconero/elasticsearch-concatenate-token-filter/tree/2.2.0">https://github.com/francesconero/elasticsearch-concatenate-token-filter/tree/2.2.0</a></td>
</tr>
<tr>
<td><strong>JVM version</strong></td>
<td>java version "1.8.0_51", Java(TM) SE Runtime Environment (build 1.8.0_51-b16), Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode)</td>
</tr>
<tr>
<td><strong>OS version</strong></td>
<td>Mac OSX 10.10.5</td>
</tr>
</tbody>
</table>
<h2 dir="auto">The problem:</h2>
<p dir="auto">I am trying to use the <code class="notranslate">dictionary_decompounder</code> filter to extract sub-words and at the same time stop other words from being stemmed/decompounded using the <code class="notranslate">keyword_marker</code> filter - the <code class="notranslate">dictionary_decompounder</code> ignores tokens marked as keywords and de-compounds them.</p>
<p dir="auto">The expectation is that the tokens marked as keywords are not touched by the <code class="notranslate">dictionary_decompounder</code> filter (in the same way as stemmer filters do not touch them).</p>
<p dir="auto">Additionally I notice that the <code class="notranslate">dictionary_decompounder</code> will generate duplicate tokens when the input token is equal to a matching sub-word ("keuken" in the example below) - this looks like it is not intended but it is easily worked around using a <code class="notranslate">unique</code> filter (so, for me, it's not so much of a problem)</p>
<h2 dir="auto">Steps to reproduce**:</h2>
<h3 dir="auto">1. create a test index:</h3>
<blockquote>
<p dir="auto"><em>N.B.</em> the <code class="notranslate">control_*</code> analyzers are included to show that stemming does not occur on keywords and also that the word "messing" is [incorrectly] stemmed to "mes"</p>
</blockquote>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
    &quot;analysis&quot;    : {
        &quot;analyzer&quot;      : {
            &quot;control_stem&quot;      : {
                &quot;type&quot;          : &quot;custom&quot;,
                &quot;tokenizer&quot;     : &quot;standard&quot;,
                &quot;filter&quot;        : [&quot;kp_snow&quot;]
            },

            &quot;control_kwstem&quot;    : {
                &quot;type&quot;          : &quot;custom&quot;,
                &quot;tokenizer&quot;     : &quot;standard&quot;,
                &quot;filter&quot;        : [&quot;nl_keywords&quot;, &quot;kp_snow&quot;]
            },

            &quot;test&quot;              : {
                &quot;type&quot;          : &quot;custom&quot;,
                &quot;tokenizer&quot;     : &quot;standard&quot;,
                &quot;filter&quot;        : [&quot;nl_keywords&quot;, &quot;nl_dehyphenate&quot;, &quot;kp_snow&quot;]
            }
        },

        &quot;filter&quot;        : {
            &quot;kp_snow&quot;           : {
                &quot;type&quot;                      : &quot;snowball&quot;,
                &quot;language&quot;                  : &quot;Kp&quot;
            },


            &quot;nl_keywords&quot;       : {
                &quot;type&quot;                      : &quot;keyword_marker&quot;,
                &quot;keywords&quot;                  : [
                    &quot;messing&quot;
                ]
            },

            &quot;nl_dehyphenate&quot;: {
                &quot;type&quot;                      : &quot;dictionary_decompounder&quot;,
                &quot;min_word_size&quot;             : 6,
                &quot;min_subword_size&quot;          : 3,
                &quot;max_subword_size&quot;          : 15,
                &quot;word_list&quot;                 : [
                    &quot;keuken&quot;,
                    &quot;mes&quot;
                ]
            }
        }
    }
}"><pre class="notranslate">{
    <span class="pl-ent">"analysis"</span>    : {
        <span class="pl-ent">"analyzer"</span>      : {
            <span class="pl-ent">"control_stem"</span>      : {
                <span class="pl-ent">"type"</span>          : <span class="pl-s"><span class="pl-pds">"</span>custom<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"tokenizer"</span>     : <span class="pl-s"><span class="pl-pds">"</span>standard<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"filter"</span>        : [<span class="pl-s"><span class="pl-pds">"</span>kp_snow<span class="pl-pds">"</span></span>]
            },

            <span class="pl-ent">"control_kwstem"</span>    : {
                <span class="pl-ent">"type"</span>          : <span class="pl-s"><span class="pl-pds">"</span>custom<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"tokenizer"</span>     : <span class="pl-s"><span class="pl-pds">"</span>standard<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"filter"</span>        : [<span class="pl-s"><span class="pl-pds">"</span>nl_keywords<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>kp_snow<span class="pl-pds">"</span></span>]
            },

            <span class="pl-ent">"test"</span>              : {
                <span class="pl-ent">"type"</span>          : <span class="pl-s"><span class="pl-pds">"</span>custom<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"tokenizer"</span>     : <span class="pl-s"><span class="pl-pds">"</span>standard<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"filter"</span>        : [<span class="pl-s"><span class="pl-pds">"</span>nl_keywords<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>nl_dehyphenate<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>kp_snow<span class="pl-pds">"</span></span>]
            }
        },

        <span class="pl-ent">"filter"</span>        : {
            <span class="pl-ent">"kp_snow"</span>           : {
                <span class="pl-ent">"type"</span>                      : <span class="pl-s"><span class="pl-pds">"</span>snowball<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"language"</span>                  : <span class="pl-s"><span class="pl-pds">"</span>Kp<span class="pl-pds">"</span></span>
            },


            <span class="pl-ent">"nl_keywords"</span>       : {
                <span class="pl-ent">"type"</span>                      : <span class="pl-s"><span class="pl-pds">"</span>keyword_marker<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"keywords"</span>                  : [
                    <span class="pl-s"><span class="pl-pds">"</span>messing<span class="pl-pds">"</span></span>
                ]
            },

            <span class="pl-ent">"nl_dehyphenate"</span>: {
                <span class="pl-ent">"type"</span>                      : <span class="pl-s"><span class="pl-pds">"</span>dictionary_decompounder<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"min_word_size"</span>             : <span class="pl-c1">6</span>,
                <span class="pl-ent">"min_subword_size"</span>          : <span class="pl-c1">3</span>,
                <span class="pl-ent">"max_subword_size"</span>          : <span class="pl-c1">15</span>,
                <span class="pl-ent">"word_list"</span>                 : [
                    <span class="pl-s"><span class="pl-pds">"</span>keuken<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>mes<span class="pl-pds">"</span></span>
                ]
            }
        }
    }
}</pre></div>
<h3 dir="auto">2. Perform an analyzer query</h3>
<p dir="auto">Assuming the created index is called "<strong>decomp_test</strong>" the following request illustrate expected/actual analyzer results:</p>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="http://localhost:9200/decomp_test/_analyze?analyzer=test"><pre class="notranslate">http://localhost:9200/decomp_test/_analyze<span class="pl-k">?</span>analyzer=test</pre></div>
<table role="table">
<thead>
<tr>
<th>Actual Tokens</th>
<th>Expected Tokens</th>
</tr>
</thead>
<tbody>
<tr>
<td>messing, mes, keukenme, keuk, mes, keuk, keuk</td>
<td>messing, keukenme, keuk, mes, keuk</td>
</tr>
</tbody>
</table>
<p dir="auto">A potentially useful processor is one that can generate one or more "fingerprints" from an incoming document.  This could aid in finding duplicates, detecting plagarism, or clustering similar documents together.</p>
<p dir="auto">I think there are two realms of fingerprinting: content fingerprinting and structural fingerprinting.</p>
<h3 dir="auto">Content Fingerprinting</h3>
<p dir="auto">Hashes the content of fields to generate a fingerprint-per-field, and optionally, a fingerprint that represents all the fields.  Could use simple hashing, or perhaps something more sophisticated like MinHash, SimHash, <a href="http://igm.univ-mlv.fr/~mac/ENS/DOC/sigmod03-1.pdf" rel="nofollow">Winnowing</a> or <a href="http://www.cs.cmu.edu/~gveda/reports/IITK/cs497_DocFingerprinting.pdf" rel="nofollow">Probabilistic Fingerprinting</a>.</p>
<p dir="auto">The API could look something like:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;fingerprint&quot;: {
    &quot;type&quot;: &quot;content&quot;,
    &quot;fields&quot;: [&quot;foo&quot;, &quot;bar&quot;],
    &quot;hash&quot;: &quot;minhash&quot;,
    &quot;hash_all&quot;: true
  }
}"><pre class="notranslate">{
  <span class="pl-ent">"fingerprint"</span>: {
    <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>content<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"fields"</span>: [<span class="pl-s"><span class="pl-pds">"</span>foo<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>bar<span class="pl-pds">"</span></span>],
    <span class="pl-ent">"hash"</span>: <span class="pl-s"><span class="pl-pds">"</span>minhash<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"hash_all"</span>: <span class="pl-c1">true</span>
  }
}</pre></div>
<p dir="auto">E.g. specify the type of fingerprinting we want to do (<code class="notranslate">content</code>), a list of fields to hash, the style of hashing and if we should also hash all the hashes together.  The output would then be the document + new fingerprint fields:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;foo&quot;: &quot;...&quot;,
  &quot;bar&quot;: &quot;...&quot;,
  &quot;fingerprint_foo&quot;: 1283891,
  &quot;fingerprint_bar&quot;: 8372038,
  &quot;fingerprint_all&quot;: 3817273
}"><pre class="notranslate">{
  <span class="pl-ent">"foo"</span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"bar"</span>: <span class="pl-s"><span class="pl-pds">"</span>...<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"fingerprint_foo"</span>: <span class="pl-c1">1283891</span>,
  <span class="pl-ent">"fingerprint_bar"</span>: <span class="pl-c1">8372038</span>,
  <span class="pl-ent">"fingerprint_all"</span>: <span class="pl-c1">3817273</span>
}</pre></div>
<h3 dir="auto">Structural Fingerprinting</h3>
<p dir="auto">The other mode of fingerprinting could be structural in nature (this is the one I'm more interested in, tbh).  Instead of fingerprinting the content of fields, we are actually fingerprinting the structure of the document itself.  Essentially, we would recursively parse the JSON and hash the keys at each level in the JSON tree.  These hashes then become a fingerprint for the structure of the document.</p>
<p dir="auto">Importantly, this type of fingerprinting ignores the leaf values...we just want to fingerprint the JSON keys themselves.</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;fingerprint&quot;: {
    &quot;type&quot;: &quot;structure&quot;,
    &quot;root&quot;: [&quot;foo&quot;],
    &quot;recursive&quot;: true,
    &quot;hash&quot;: &quot;murmur3&quot;,
    &quot;hash_all&quot;: true
  }
}"><pre class="notranslate">{
  <span class="pl-ent">"fingerprint"</span>: {
    <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>structure<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"root"</span>: [<span class="pl-s"><span class="pl-pds">"</span>foo<span class="pl-pds">"</span></span>],
    <span class="pl-ent">"recursive"</span>: <span class="pl-c1">true</span>,
    <span class="pl-ent">"hash"</span>: <span class="pl-s"><span class="pl-pds">"</span>murmur3<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"hash_all"</span>: <span class="pl-c1">true</span>
  }
}</pre></div>
<ul dir="auto">
<li><code class="notranslate">root</code>: defines where to start recursing, in case you only care about a portion of the document.  Could be omitted or set to <code class="notranslate">"*"</code> to process the entire document</li>
<li><code class="notranslate">recursive</code> if you want the processor to fingerprint all the layers.  False if you just want the top-level of keys hashed.</li>
<li><code class="notranslate">hash</code>: murmur, minhash, etc</li>
<li><code class="notranslate">hash_all</code>: if all the hashes should be hashed together to build a final fingerprint</li>
</ul>
<p dir="auto">And the new document:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
  &quot;foo&quot;: {
    &quot;bar&quot;: {
      &quot;baz&quot;: &quot;buzz&quot;
    },
    &quot;beep&quot;: {
      &quot;boop&quot;: &quot;bop&quot;
    }
  },
  &quot;fingerprint_level1&quot;: 001734,
  &quot;fingerprint_level2&quot;: 992727,
  &quot;fingerprint_level3&quot;: 110293,
  &quot;fingerprint_all&quot;: 235240
}"><pre class="notranslate">{
  <span class="pl-ent">"foo"</span>: {
    <span class="pl-ent">"bar"</span>: {
      <span class="pl-ent">"baz"</span>: <span class="pl-s"><span class="pl-pds">"</span>buzz<span class="pl-pds">"</span></span>
    },
    <span class="pl-ent">"beep"</span>: {
      <span class="pl-ent">"boop"</span>: <span class="pl-s"><span class="pl-pds">"</span>bop<span class="pl-pds">"</span></span>
    }
  },
  <span class="pl-ent">"fingerprint_level1"</span>: <span class="pl-ii">001734</span>,
  <span class="pl-ent">"fingerprint_level2"</span>: <span class="pl-c1">992727</span>,
  <span class="pl-ent">"fingerprint_level3"</span>: <span class="pl-c1">110293</span>,
  <span class="pl-ent">"fingerprint_all"</span>: <span class="pl-c1">235240</span>
}</pre></div>
<p dir="auto">Instead of a fingerprint-per-field, we now have one per "level".</p>
<p dir="auto">I can think of a number of objections to both of these, but this should at least kick off the discussion :)</p>
<p dir="auto">/cc <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/martijnvg/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/martijnvg">@martijnvg</a></p>